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ABSTRACT 


U.S.  Navy  Submarine  Approach  Officers  (AO)  use  tactical  skills  acquired  from 
years  at  sea  and  a  seemingly  natural  panache  to  accurately  determine  an  enemy 
submarine’s  bearing,  range,  course,  and  speed.  This  thesis  investigates  the  effects  of  AO 
demographic  differences  and  combat  system  employment  methodologies  on  the  ability  to 
develop  a  timely  and  accurate  firing  solution.  Employing  a  low-resolution  submarine 
combat  simulator,  approximately  10  percent  of  the  total  pool  of  AO’s  were  taken  through 
two  of  four  pre-scripted  initial  contact  scenarios.  The  AO’s  were  instructed  to  execute 
each  scenario  until  they  perceive  that  an  accurate  solution  is  obtained  on  the  enemy 
submarine.  The  demographic  differences  of  geographic  location,  ship  type,  and  duty  type 
are  the  top  three  traits  that  distinguish  between  success  and  failure.  Further  data  analysis 
reveals  differences  in  the  information  utilization  of  the  simulator  between  successful  and 
unsuccessful  operators.  Additionally,  utilizing  a  survey  administered  to  the  subjects  and 
basic  display  design  principles,  a  notional  command  workstation  for  the  next  generation 
of  submarine  is  developed.  The  conclusions  of  this  research  provide  insight  into  the 
cognitive  modeling,  training,  and  selection  of  AO’s,  as  well  as  adding  to  the  growing 
body  of  work  in  the  design  of  military  decision  support  systems. 
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DISCLAIMER 


The  views  expressed  in  this  thesis  are  those  of  the  author  and  do  not  reflect  the 
official  policy  or  position  of  the  Department  of  Defense  or  the  U.S.  Government. 

The  reader  is  cautioned  that  the  computer  programs  developed  in  this  research 
may  not  have  been  exercised  for  all  cases  of  interest.  While  every  effort  has  been  made, 
within  the  time  available,  to  ensure  that  the  programs  are  free  of  computational  and 
logical  errors,  they  cannot  be  considered  validated.  Any  application  of  these  programs 
without  additional  verification  is  at  the  risk  of  the  user. 

The  simulator  utilized  in  this  research  was  developed  by  the  Naval  UnderSea 
Warfare  Center  (NUWC).  Any  inquires  concerning  its  origination  or  use  should  be 
directed  to  NUWC. 
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EXECUTIVE  SUMMARY 


Recent  changes  in  U.S.  Naval  doctrine  have  underscored  the  need  for  Navy  ships 
to  operate  in  the  littoral  regions  of  the  world.  Operating  in  proximity  to  congested  coastal 
regions  present  unique  challenges  to  the  military  decision-maker.  Frequently,  there  is  no 
clear-cut  answer  about  a  specific  situation  or  decision.  Rapidly  unfolding  events  result  in 
severe  time  pressure  and  extreme,  often  fatal,  consequences.  Current  real-time  battle- 
space  management  systems  are  well  suited  to  the  demands  of  a  protracted  blue  water 
conflict.  However,  their  ability  to  be  optimally  employed  in  the  emergent  fast-moving 
littoral  environment  is  questionable. 

Project  NEMO  (Naval  Expertise  and  Modeling)  was  initiated  by  the  Naval 
UnderSea  Warfare  Center  (NUWC)  to  explore  Combat  Control  System  (CCS) 
modifications  to  best  meet  Submarine  Approach  Officers  (AO)  needs  in  these  new 
environments.  Project  NEMO’s  ultimate  goal  is  “to  provide  for  the  layout,  design,  and 
implementation  of  a  Submarine  Command  Workstation  (CWS)  to  be  incorporated  into 
the  New  Attack  Submarine  (NSSN)”.  Utilizing  a  small  cadre  of  submarine  officers 
stationed  at  NUWC,  Project  NEMO  researchers  are  currently  in  the  process  of  modeling 
the  AO's  cognitive  processes  during  tactical  engagements.  While  the  initial  CCS  for  the 
NSSN  is  in  the  final  stages  of  development,  it  is  postulated  that  lessons  learned  from 
Project  NEMO  will  influence  future  upgrades  and  provide  for  system  improvements. 

This  thesis  has  three  primary  objectives.  The  first  objective  is  to  explore  possible 
relationships  between  both  subject  demographic  differences  and  CCS  employment 
strategies  in  the  prediction  of  a  successful  engagement.  The  second  objective  is  to 
provide  data  in  the  development  of  a  notional  "AO  friendly"  CWS  interface.  It  is 
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postulated  that  this  interface  will  allow  the  AO  to  more  accurately  evaluate  relevant 
tactical  information  with  less  effort  through  the  use  of  more  appropriate  information 
presentation  and  display  formatting.  The  third  objective  is  to  provide  for  principles 
which  could  be  applied  in  the  areas  of  personnel  selection,  qualification,  and  training. 

A  low-resolution  computer-based  CCS  simulation  was  used  to  run  subjects 
though  two  of  four  pre-scripted  initial  contact  scenarios  in  a  modified  Latin  square 
design.  The  subject  pool  comprised  approximately  10%  of  the  active  duty  U.S.  AO’s. 
The  study  used  two  objective  measures  of  effectiveness,  solution  accuracy  at  time  of  fire 
(TOF)  and  weapon  to  target  closest  point  of  approach  (CPA),  and  one  subjective  measure 
of  effectiveness;  the  determination  of  a  “successful”  engagement.  In  addition,  subjects 
participated  in  a  design  survey  to  provide  feedback  on  not  only  the  simulation,  but  also 
their  desires  for  the  next  generation  CWS. 

An  exploratory  statistical  analysis  was  performed  on  the  subject-generated 
database  with  the  goal  of  determining  if  demographic  distinctions  and  CCS  usage 
strategies  among  AO’s  could  be  used  to  predict  the  quality  of  a  simulated  engagement. 
Regression  techniques  were  then  applied  to  the  MOE’s  obtained  for  the  subjects. 
Geographical  location,  ship  type,  and  duty  type  were  the  three  most  important 
determinants  in  the  prediction  of  a  successful  engagement.  Surprisingly,  no  distinction 
between  experts  and  novice  operators  was  found  with  the  MOE's.  Coupling  this  finding 
with  an  almost  zero  correlation  between  accuracy  and  number  of  years  at  sea,  a  hallmark 
for  expert  designation,  raises  the  question  as  to  whether  a  good  AO  is  "bom  or  made". 
Analysis  of  the  linear  relationships  between  TOF  and  CPA  revealed  two  important 
observations:  1)  the  shorter  the  engagement,  the  more  accurate  target  solution,  and  2) 
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expert  vs.  novice  distinction  is  negligible,  while  geographical  location,  ship  type,  and 
education  are  significant  factors  in  predicting  success. 

A  single  step  link  analysis  was  examined  on  both  display  usage  and  solution 
generation.  Rather  than  examining  the  differences  between  expert  and  novice  AO’s,  the 
differences  between  successful  and  unsuccessful  AO’s  were  used  instead.  It  was  shown 
that  successful  and  unsuccessful  subjects  do  in  fact  demonstrate  variance  in  the  cognitive 
progression  of  links  in  both  display  usage  and  solution  generation.  The  successful 
subjects  were  shown  to  follow  standard  submarine  liturgy  and  submerged  approach 
methodology  while  unsuccessful  subjects  were  observed  to  be  acting  in  an  ad  hoc  manner 
(i.e.  "easter  egging”).  This  further  reinforces  the  benefit  of  reliance  on  established 
methodologies  and  techniques  and  provides  for  applications  in  the  periodic  reevaluation 
and  training  of  AO's. 

Results  from  the  two  link  analyses  were  contrasted  with  both  subject  evaluations 
of  the  current  CCS’s  and  subject  desires  for  CCS  attributes  into  a  preliminary  CWS 
design.  While  originally  it  was  envisioned  subjects  would  only  "brainstorm"  the  CWS 
fire  control  module,  subject  dissatisfaction  with  current  CCS's  and  the  desire  for  a 
revolutionary  leap  in  its  employment,  significantly  expanded  the  scope  of  subject  inputs. 
The  final  product  can  be  viewed  as  a  notional  design  concept  for  a  complete  CWS. 

The  main  value  of  this  study  is  the  continued  exploration  of  the  cognitive 
processes  of  the  submarine  AO.  The  results  obtained  have  implications  in  initial 
selection,  periodic  reevaluation,  career  progression,  and  training  regimen  of  personnel. 
The  data  collected  in  the  course  of  this  research  will  continue  to  provide  additional 
insight  and  future  work  should  continue  to  explore  the  hypothesis  not  yet  examined. 
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I.  INTRODUCTION 


Changes  in  United  States  strategic  priorities  have  generated  an  increased  need  for 
U.S.  Navy  ships  to  operate  in  the  littoral  regions  of  the  world  (Department  of  the  Navy, 
1996).  Naval  doctrine  specifies  that  the  near  term  battle-space  will  be  exclusively  in  the 
littorals.  Littoral  regions,  defined  as  the  area  within  200  nautical  miles  of  the  coastline, 
necessitate  operations  in  the  congested,  confined  water  and  airspace  close  to  an 
adversary’s  shores  (Hutchins,  1995).  Littoral  warfare  presents  unique  challenges  to  the 
decision-maker  for  it  entails  multi-faceted  scenarios  characterized  by  rapidly  unfolding 
events  involving  contact  identification,  comprehension  of  intent,  consideration  of 
available  responses,  and  forecasting  potential  consequences.  Frequently,  there  are  no 
clear-cut  answers  about  specific  situations  and  often  these  events  can  have  extreme,  even 
fatal,  consequences  for  errors  (Collyer  &  Malecki,  1998).  Current  real-time  battle-space 
management  systems  are  well  suited  to  the  demands  of  protracted  blue  water  conflicts; 
however,  their  viability  in  the  littoral  environment  is  questionable. 

A.  BACKGROUND 

There  have  been  several  well-publicized  and  unfortunate  events  that  have  focused 
the  military’s  attention  on  the  difficult  decisions  facing  U.S.  Naval  commanders  in  the 
littorals.  The  1987  USS  STARK  (Levinson,  1997)  and  the  1988  USS  VINCENNES 
(Rogers,  1996)  incidents  are  still  fresh  in  the  Navy’s  collective  consciousness.  In  the 
case  of  the  STARK,  a  harassing  Iraqi  F-4  jet  was  allowed  to  shoot  two  anti-ship  missiles, 
severely  damaging  the  ship  and  tragically  resulting  in  37  crew  fatalities.  This  situation 
was  predicated  by  the  confusing  and  contradictory  rules  of  engagement  (ROE)  under 
which  the  STARK  was  operating.  In  the  case  of  the  VINCENNES,  an  Iranian 
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commercial  jetliner  carrying  288  civilians  was  mistakenly  identified  as  an  inbound 
hostile  aircraft,  and  was  consequently  shot  down.  The  official  after  action  report  on  the 
VINCENNES  incident  found  that  the  decision  making  process  was  influenced  by  the 
events  surrounding  the  STARK  the  year  before.  While  circumstances  leading  up  to,  and 
surrounding  the  two  events  are  very  different,  in  both  cases  the  basic  mistake  remains  the 
same.  The  Commanding  Officer  (CO),  operating  under  conditions  of  extreme  stress, 
made  the  wrong  decision  about  whether  of  not  to  engage  an  inbound  aircraft. 

In  recognition  of  the  complex  and  difficult  decisions  that  led  up  to  the  STARK 
and  VINCENNES  incidents,  the  Tactical  Decision-Making  Under  Stress  (TADMUS) 
program  was  initiated  (See  Appendix  A  for  a  full  discussion  of  TADMUS).  The 
objective  of  the  TADMUS  program  was  to  explore  the  areas  of  human  factors  and 
training  technology  in  order  to  develop  and  apply  principles  that  could  help  avoid  such 
incidents  in  the  future  (Bowers  &  Salas,  1998).  TADMUS  strives  to  act  as  an  intelligent 
assistant  to  the  CO/Tactical  Action  Officer  (TAO)  team  of  an  AEGIS  cruiser  during 
periods  when  the  mental  workload  exceeds  the  teams  combined  cognitive  capacity.  It 
enhances  the  visual  stimulus  presented  to  the  team  while  simultaneously  overcoming 
their  cognitive  limitations  in  support  of  a  recognition  primed  heuristic  for  the  current 
tactical  scenario.  Furthermore,  it  attempts  to  model  the  human  cognitive  decision  making 
process  by  independently  forming  hypotheses  concerning  target  motivation  and 
subsequent  own  ship  response  options.  At  the  time  of  this  research,  TADMUS  was  in  the 
process  of  undergoing  final  approval  prior  to  implementation  aboard  AEGIS  cruisers 
(Morrison,  Kelly,  Moore,  &  Hutchins,  1998). 
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There  are  several  “submerged  events"  similar  to  the  STARK  and  VINCENNES 
incidents  that  currently  weigh  heavily  on  the  U.S.  Navy  Submarine  community.  The 
classification  of  this  paper  does  not  permit  a  frank  discussion  of  such  incidents,  but  it 
suffices  to  say  that  such  incidents  do  occur  (Bowermaster,  1993;  Clancy,  1997).  In  fact, 
only  recently  has  the  1993  underwater  collision  between  the  USS  GRAYLING  and  a 
Soviet  Delta  class  SSBN  in  the  North  Sea  been  made  public.  The  Submarine  community 
has  realized  the  potential  benefits  of  a  TADMUS-like  system  and  is  eagerly  pursuing  the 
acquisition  of  such  a  system. 

The  Naval  UnderSea  Warfare  Center  (NUWC)  initiated  Project  NEMO  (Naval 
Expertise  and  Modeling)  in  the  mid-1990's  to  examine  possible  changes  for  the  current 
Combat  Control  System  (CCS)  that  best  meet  the  needs  of  the  Submarine  Approach 
Officers  (AO)  during  a  tactical  engagement.  Project  NEMO’s  ultimate  goal  is  “to 
provide  for  the  layout,  design,  and  implementation  of  a  Submarine  Command 
Workstation  (CWS)  to  be  incorporated  into  the  New  Attack  Submarine  (NSSN)” 
(Kirschenbaum,  1997).  Project  NEMO  researchers  are  currently  modeling  the  AO's 
cognitive  processes  during  tactical  engagements  based  on  a  small  a  small  cadre  of 
Submarine  officers  stationed  at  NUWC  (Kirschenbaum,  1997).  However,  the  program  is 
still  in  the  infancy  stage,  with  many  years  of  difficult  work  ahead.  While  the  initial  CCS 
for  the  NSSN  is  currently  in  its  final  stages  of  development,  it  is  hoped  that  Project 
NEMO's  findings  will  influence  future  upgrades  and  provide  for  substantial 
improvements. 
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B.  OBJECTIVE  STATEMENT 

By  understanding  and  capitalizing  upon  the  AO’s  cognitive  processes,  the  CWS 
could  allow  for  a  more  accurate  evaluation  of  relevant  tactical  information  with  less 
effort  as  compared  to  existing  tactical  displays.  Thus,  this  thesis  has  two  primary 
objectives: 

1)  To  assess  the  relationship  between  both  AO  demographic  variables,  such  as 
rank,  experience,  education,  etc.,  and  AO  CCS  employment  strategies  to  the 
probability  of  a  "successful"  tactical  engagement.  These  variables  and 
strategies  could  then  be  used  as  weighting  factors  in  the  areas  of  personnel 
selection  and  qualification. 

2)  To  develop  a  basic  design  concept  for  an  "AO  friendly"  CWS  interface  based 
on  the  CCS  informational  usage,  display  employment  frequency,  and  AO 
subjective  inputs.  This  interface  would  be  the  centerpiece  for  an  AO  decision 
support  system.  This  system  could  also  be  used  as  a  training  aid  during 
mission  preparation. 

C.  STATEMENT  OF  THE  PROBLEM 

Cognitive  task  analysis  of  the  AO  and  their  goal  allocation  structure  has  been 
examined  in  detail  as  an  initial  part  of  Project  NEMO  (Kirschenbaum,  1997).  The  next 
stage  focuses  on  what  accounts  for  the  differences  between  "successful"  and 
"unsuccessful"  AO’s.  Specifically,  the  questions  to  be  answered  are:  are  the  mental 
models  of  AO’s,  as  they  relate  to  informational  usage  from  the  CCS,  the  same  and  do 
successful  and  unsuccessful  AO’s  share  the  same  level  of  situational  awareness 
throughout  the  course  of  a  tactical  scenario?  It  is  thought  that  the  AO's  CCS  usage 


strategy  and  success  rate  might  be  related  to  their  status  as  either  an  "Expert"  or  "Novice" 
AO.  Demographic  variables  such  as  current  job,  ship  type,  location  stationed,  general 
computer  experience,  combat  system  experience,  and  post-secondary  education  might  be 
also  be  factors  that  contribute  in  the  distinction  between  CCS  usage  strategies  and  the 
prediction  of  success. 

Currently,  AO’s  have  no  display  specifically  designed  to  give  them  the  “big 
picture”  of  an  engagement.  Figure  1.1  shows  the  typical  tactical  aid  that  submarine  AO’s 
use  to  plan  and  execute  attacks.  The  tactical  aid  is  actually  an  8-inch  square  piece  of 
plexi-glass  covering  a  standard  naval  maneuvering  board.  The  writing  on  the  tactical  aid 
comes  from  a  black  grease  pencil  used  by  the  AO.  Information  such  as  own-ship’s 
current  and  past  course,  speed,  and  depth,  current  sub-surface  and  surface  environmental 
conditions,  ancillary  contact  parameters,  as  well  as  pertinent  contact  kinematics  are 
typically  detailed  on  the  tactical  aid.  It  should  be  noted  though,  that  there  is  neither  a 
defined  set  of  requirements  for  what  information  that  should  be  placed  on  the  tactical  aid 
nor  a  prescribed  format  for  that  information.  It  is  simply  a  matter  of  personal  choice. 
Additionally,  in  order  to  construct  this  tactical  aid,  the  AO  compiles  information  from  up 
to  fourteen  distinct  displays,  five  individuals,  and  three  separate  locations  within  the 
submarine  control  area.  Once  gathered,  the  data  is  manually  correlated  for  each  target  in 
the  current  situation,  and  then  all  targets  are  correlated  to  achieve  the  big  picture. 
Compounding  this  situation,  the  submarine  control  area  typically  has  30-45  men  in  a  very 
confined  space  of  as  little  as  25  x  25  feet.  Such  environmental  distracters  only  serve  to 
degrade  a  commander’s  already  reduced  capability  to  successfully  execute  his  primary 
task.  This  is  an  unacceptable  situation,  and  therefore  an  attempt  will  be  made  to  develop 
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a  notional  submarine  command  level  display  concept  that  provides  for  more  complete 
utilization  of  information  and  an  increased  situational  awareness  (Kirschenbaum,  1997). 


Figure  1.1  Typical  AO  Tactical  Aid. 


D.  SCOPE,  LIMITATIONS,  AND  ASSUMPTIONS 

The  present  effort  is  confined  to  the  evaluation  of  active  duty  U.S.  Naval 
Submarine  Officers  in  the  use  of  a  simulated  CCS.  Due  to  the  simulator's  low  resolution, 
limits  imposed  on  scenario  run  time,  subject  workload  concerns,  and  military 
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classification,  some  realistic  elements  were  impossible  to  incorporate.  All  subjects  are 
assumed  to  be  willing  participants  whose  motivation  is  driven  by  pride  and 
professionalism. 

This  paper  is  organized  into  five  sections.  First,  the  motivation  for  and  objective 
of  this  research  is  described.  Next,  an  in  depth  review  of  the  literature  supporting  the 
objectives  is  presented.  The  methodology,  which  comprises  both  the  experiment 
configuration  and  outcome  generation,  is  described  in  the  third  section.  The  fourth 
section  contains  the  results  of  exploratory  analysis  performed  on  the  subject-generated 
database.  The  final  section  discusses  the  results  and  provides  lessons  learned  and 
recommendations  for  follow  on  work. 
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II.  REVIEW  OF  THE  LITERATURE 


Edwards  (1988)  defines  Human  Factors  (HF)  as  the  “technology  concerned  to 
optimize  the  relationships  between  people  and  their  activities  by  the  systematic 
application  of  the  human  sciences,  integrated  within  the  framework  of  human 
engineering.”  It  attempts  to  integrate  complex  human  performance  concepts  with 
statistical  and  stochastic  methods  to  adequately  capture  the  "man-machine"  relationship. 
Because  of  the  ambiguity  involved,  the  study  of  HF  has  not  gained  universal  acceptance. 
However,  its  benefits  and  contributions  over  the  last  50  years  cannot  be  disputed 
(Lederer,  1988).  To  provide  a  common  core  of  HF  terminology  and  concepts  to  the 
reader,  a  brief  overview  of  Human  Information  Processing,  Naturalistic  Decision- 
Making,  Situational  Awareness,  Schema  Theory,  Expert  vs.  Novice  Performance, 
Display  Design,  and  Link  Analysis  Techniques  is  presented.  The  military  applicability  of 
these  areas  is  then  used  to  address  the  core  issues  of  this  research. 

A.  HUMAN  FACTORS  CONCEPTUAL  OVERVIEW 

1.  Human  Information  Processing 

Human  Information  Processing  (HIP)  is  the  vehicle  by  which  people  decipher 
data.  A  principle  feature  of  HIP  models  is  the  assumption  that  a  series  of  stages  or 
mental  operations  occur  between  external  stimuli  and  self-directed  responses.  Much  HF 
research  is  directed  at  isolating,  and  then  characterizing  each  of  these  stages.  Attributes 
such  as  capacity,  duration,  and  representational  forms  of  the  particular  stage  are  studied. 
While  the  end  result  remains  elusive,  there  are  several  generally  accepted  models  with  the 
most  widely  accepted  one  being  the  Wickens  and  Flech  (1988)  Attention  Resource  Model 
(see  Figure  2. 1). 
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attention  Resources 


Figure  2. 1  Attention  Resource  Model  (Wickens  &  Flech,  1988). 

The  first  stage  of  the  Attention  Resource  Model,  as  depicted  in  Figure  2.1,  is  the 
Sensory  Store.  It  is  generally  assumed  that  there  is  a  separate  sensory  store  for  each 
sensory  modality.  In  this  store,  the  externally  generated  physical  energy  is  transformed 
into  internal  neural  energy  and  is  represented  in  terms  of  its  salient  physical  features. 
This  information  lasts  only  a  brief  time  and  does  not  require  any  attentional  resources. 
While  the  first  stage  is  relatively  simple  and  uncomplicated,  the  second  stage,  Pattern 
Recall,  is  both  the  most  crucial  and  least  understood  of  all  of  the  stages.  It  is  at  this  stage 
that  the  physical  stimulation  in  the  sensory  stores  is  integrated  into  meaningful  elements. 
For  example,  the  dark  curved  lines  on  this  piece  of  paper  are  processed  as  meaningful 
letters  and  words.  This  process  is  very  complex  and  is  limited  in  its  execution  by  the 
pool  of  available  attentional  resources.  These  patterns  are  typically  stored  in  long  term 
memory  for  later  recall. 


The  HEP  Attention  Resource  Model  contains  two  distinct  ways  to  store 
information:  Working  Memory,  and  Long-term  Memory.  Working  memory  represents 
the  information  currently  being  used  by  the  information  processor.  The  capacity  of 
working  memory  is  limited  in  both  its  scope  and  size.  Long-term  memory  represents 
information  available  to  the  information  processor,  but  not  currently  in  use.  Unlike 
working  memory,  long-term  memory  capacity  is  essentially  unlimited. 

The  third  stage.  Decision  Selection,  is  where  a  stimulus  is  recognized  and  a 
decision  must  be  made  regarding  that  stimulus.  Various  options  are  available  at  this 
point:  the  information  can  be  stored  for  use  at  some  future  date,  it  can  be  integrated  with 
other  available  information,  or  it  may  initiate  a  response.  Each  of  these  options  generates 
their  own  associated  costs  and  benefits  that  must  be  considered  when  choosing  a  single 
course  of  action.  The  decision  generated  in  this  stage  is  finally  passed  onto  the  last  stage, 
Response  Execution.  In  this  stage,  the  selected  decision  is  translated  into  a  series  of 
unconscious  and  conscious  motor  actions  and  sequences.  The  resulting  outcomes,  by 
way  of  a  feedback  loop,  then  become  input  to  the  sensory  stores,  which  in  turn  can  be 
interpreted  and  entered  as  relevant  data  in  selecting  the  next  response.  As  with  the  second 
stage,  these  final  two  stages  are  limited  in  their  scope  by  the  available  attentional 
resources  (Kantowitz  &  Sorkin,  1983). 

2.  Naturalistic  Decision  Making 

In  the  early  1990’s,  there  was  a  radical  shift  in  the  perception  of  human  decision¬ 
making  (Collyer  &  Malecki,  1998).  The  analysis  of  actual  decision  strategies  that 
humans  use  in  solving  complex,  real-world  conditions  was  termed  “Naturalistic  Decision 
Making”  (NDM)  (Klein,  1993).  “Naturalistic”  psychologists  assert  that  expert  decision- 
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makers  rarely  use  the  traditional  resource  intensive  strategies  of  classical  decision¬ 
making  in  order  to  make  decisions  in  the  presence  of  complex,  adverse,  and  time 
constrained  scenarios.  Rather,  experts  rely  on  internal  heuristics  formed  by  the  very 
nature  of  being  recognized  system  experts  (Kozlowski,  1998).  It  is  postulated  that 
experts  tend  to  recognize  scenarios  based  on  how  closely  the  current  situation  matches 
one  previously  experienced  (Kaempf  &  Militelo,  1992).  The  expert  then  recalls  the 
previous  actions  taken  in  the  stored  scenario,  evaluates  those  actions  against  the  current 
situation,  makes  appropriate  modifications  within  the  current  time  constraints,  and  then 
activates  the  schema  containing  the  actions.  Rather  than  employing  an  optimization 
routine,  the  schema's  activated  typically  pursue  a  satisficing  strategy  (Klein,  1993; 
Hutchins,  1995;  Kirschenbaum,  1997).  In  other  words,  the  expert  selects  a  course  of 
action  which  is  not  necessarily  the  best,  but  is  still  reasonable  and  will  result  in  a  outcome 
that  is  within  the  constraints  of  the  situation  at  hand. 

There  are  two  models  of  decision-making  currently  used  in  “naturalistic”  settings, 
Recognition  Primed  Development  (RPD)  and  Explanation  Based  Reasoning  (EBR) 
(Klein,  1993).  RPD,  more  commonly  known  as  feature  matching,  “occurs  when  the 
decision  maker  recognized  the  features  of  the  present  situation  as  similar  or  identical  to 
those  of  a  previous  situation”  (Kaempf  &  Militelo,  1992).  An  unconscious  correlation 
then  triggers  the  recall  of  all  pertinent  information  learned  about  that  particular  situation: 
clues,  cues,  goals,  expectations,  plans  of  action,  and  outcomes  that  were  observed  in  the 
past.  Therefore,  it  is  thought,  expert  decision-makers  rely  on  detailed  heuristics  stored  in 
long  term  memory  to  control  decision  making  in  similar,  but  different  situations 
(Hutchins,  1996).  EBR,  more  commonly  known  as  story  generation,  occurs  when  the 
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decision-makers  find  themselves  in  complex  situations  without  all  the  required 
information,  the  information  is  self-contradictory,  and/or  certain  correlating  events  are 
absent  (Bowers,  1998).  The  decision-maker  then  develops  “neural  links”,  or  causal 
relationships,  between  facts  to  produce  a  working  representation  of  the  situation  (Klein, 
1993).  Once  constructed,  the  new  model  of  the  situation  is  acted  upon  by  the  decision¬ 
maker  within  the  current  environmental  conditions. 

Today,  advanced  Decision  Support  Systems  (DSS)  involved  in  such  industries  as 
nuclear  power  plants,  off-shore  oil  platforms,  and  emergency  service  providers  utilize  a 
mixture  of  displays  that  support  both  RPD  and  EBR  cognitive  processing  (Wickens, 
1992).  These  displays,  respectively,  are  termed  “status”  and  “command”.  Status  displays 
are  characterized  by  their  inclusion  of  “why”  information  and  informs  the  user  of  what  is 
known  and  assumed  about  the  current  situation.  Status  displays  allow  the  user  to 
examine  differing  hypotheses  without  committing  to  a  specific  course  of  action.  This  is 
in  contrast  to  the  “command”  type  display,  which  present  information  in  list  or  bullet 
format  and  tells  the  user  what  to  do  without  explaining  the  motivation  behind  the  actions. 
This  type  of  display  is  most  often  utilized  in  situations  when  quick  responses  are  required 
of  inexperienced  or  heavily  tasked  operators.  The  most  common  display  type  used  for 
military  decision  support  systems  in  the  command  display  (Irizarry  &  Knapp,  1986). 
However,  research  indicates  that  by  providing  decision  support  in  the  form  of  a 
combination  of  command  and  status  displays,  an  increase  in  the  overall  accuracy  in  the 
understanding  of  system  and  environmental  states  occurs  (Kozlowski,  1998). 
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3.  Situational  Awareness 


Enhancement  in  the  understanding  of  system  and  environmental  states  is  referred 
to  as  Situational  Awareness  (SA).  For  all  current  accepted  models  of  NDM,  SA  is  the 
sum  and  substance:  sizing  up  the  situation,  understanding  a  scenario,  defining  the 
problem,  categorizing  the  circumstances,  constructing  a  representation,  making  mental 
models,  painting  a  picture,  or  creating  images  (Federico,  1995).  Decision-making  in 
realistic  settings  is  a  process  of  not  only  producing  and  updating  a  representation  of  the 
perceived  situation,  but  also  specifying  and  appraising  possible  actions. 

Unfortunately,  perception,  action  appraisal,  and  mental  choices  are  difficult  to 
measure.  However,  since  SA  is  the  crux  of  user  comparison,  measures  of  effectiveness 
(MOE's)  must  be  obtained  (Federico,  1995).  Typically  in  military  settings,  two 
approaches  are  taken  in  the  measurement  of  SA.  The  first  is  the  measurement  of  critical 
factors  or  preconditions  that  lead  to,  or  inhibit  SA.  By  the  measurement  of  two 
secondary  load  factors,  attention  and  working  memory,  the  acquisition  and  interpretation 
of  information  from  the  environment  necessary  to  form  S  A  can  be  ascertained  (Endsley, 
1995).  The  second  is  the  attainment  of  an  objective  MOE.  This  is  typically  measured  in 
terms  the  military  mind  easily  understands:  minimum  time  or  range,  hit  or  miss,  or  life  or 
death. 

Of  the  current  NDM  models  in  use  today,  some  link  S  A  to  the  selection  of  action 
alternatives  while  others  indicate  that  SA  precedes  and  initiates  the  selection  of  action 
alternatives.  Hammond’s  (1986)  Intuitive  Decision,  Lipshitz’s  (1989)  Matching  Mode 
Decisions,  and  Rasmussen’s  (1983)  Skill-based  and  Rule-based  behaviors  are  all 
examples  in  which  the  perceived  situation  directly  determines  the  chosen  action. 
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Alternatively,  Beach’s  (1990)  Framing  Action  Selection  Criteria,  Connolly’s  (1982) 
Cognitive  Mapping,  and  Klein’s  (1990)  model  of  Critical  Decision  Making  (CDM)  are 
examples  in  which  recognizing  the  situation  precedes  the  determination  of  the  chosen 
alternative. 

NDM  models  as  a  group  tend  to  emphasize  certain  salient  cognitive  processes  that 
are  associated  with  SA.  Representing  problems  using  knowledge  structures  or  schema 
(Connolly,  1982;  Beach,  1990),  categorizing  situations  (Rasmussen,  1983;  Klein,  1989; 
Lipshitz,  1989),  and  using  storytelling  or  mental  modeling  (Hammond,  1986;  Lipshitz, 
1989;  Beach,  1990)  are  some  of  the  different  methods  espoused.  However,  the  common 
thread  running  though  all  of  these  models  is  that  the  knowledge  necessary  to  assess  and 
recognize  situations  for  NDM  is  incorporated  into  stored  schema  (Federico,  1995). 

4.  Schema  Theory 

Schemas  are  commonly  defined  as  stored  metal  representations  of  previous 
experiences  (Bailey,  1980).  In  other  words,  a  schema  is  like  an  organized  memory  unit, 
similar  to  a  stored  computer  program.  These  memory  units  can  be  stored  and 
subsequently  recalled  using  any  of  the  five  senses  (Wickens,  1994).  Once  invoked,  a 
schema  is  typically  executed  by  a  series  of  automatic  and  unconscious  motor  action 
sequences.  For  example,  starting  your  car  and  driving  to  work,  making  toast  for 
breakfast,  or  calling  relatives  on  the  telephone,  are  all  examples  of  everyday  tasks  most 
people  accomplish  in  an  unconscious  manner.  An  informal  representation  of  the  schema 
theory  model  is  shown  in  Figure  2.2. 
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Concious  Thought 


Unconcious  Thought 


Figure  2.2  Schema  Theory  Model  (NPGS  School  of  Aviation  Safety  Handout). 

In  the  current  context  of  NDM,  it  is  logical  to  infer  that  it  is  the  stored  schema  that 
drives  decisions.  Consequently,  this  schema-driven  decision  making  is  primarily  a  RPD 
process.  Stated  differently,  decision-making  is  comprised  of  recognizing  previous 
scenarios  and  situations,  the  courses  of  action  taken,  and  the  eventual  outcomes  (Zachary, 
Ryder  &  Hicinbothom,  1998).  It  is  postulated  that  entire  patterns  and  sequences  of 
events  are  employed  in  their  entirety  to  facilitate  recognition  and  assessment  of  similar 
scenarios.  When  similar  situations  are  recognized  or  experienced,  stored  schema  is 
activated,  and  past  successful  actions  are  applied  to  the  present  situation.  This  discussion 
is  predicated  on  the  notion  that  the  decision-maker  in  fact,  possesses  the  necessary 
schema  in  the  form  of  neural-networked  identification,  action,  and  solution  links.  This 
assumption  of  correct  mental  representation  and  its  correct  employment,  is  the  primary 
distinguishing  trait  of  the  so-called  expert  decision-maker.  (Gick  &  Holyoak,  1983; 
Beach,  1990;  Wickens,  1992). 
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5.  Expert  vs.  Novice  Performance 

The  core  elements  of  expert  vs.  novice  distinction  and  subsequent  performance 
implications  have  been  the  subject  of  numerous  investigations  (Federico,  1995;  Glaser, 
1985;  Klein,  1993;  Wickens,  1992).  These  observations  have  been  drawn  from  the 
analysis  of  decision-makers  that  routinely  operate  under  conditions  of  extreme  time 
constraints  and  stress:  emergency  room  doctors,  aircraft  pilots,  nuclear  plant  operators, 
etc.  When  taken  as  a  group,  the  observations  of  previous  researchers  seem  to  share 
several  important  aspects:  1)  experts  tend  to  excel  primarily  because  they  possess 
extensive  knowledge  in  their  respective  areas,  2)  experts  distinguish  patterns  in  task 
execution  that  indicate  superior  knowledge  organization  skills,  3)  experts  perform  tasks 
quicker  with  less  errors,  4)  experts  process  superior  STM  and  LTM,  5)  experts  perceive 
and  categorize  situations  at  a  deeper  and  more  principled  level,  6)  experts  initiate  and 
maintain  correct  mental  representations  of  the  situation  and  inherent  constraints  at  hand, 
7)  experts  posses  superior  self-monitoring  skills  when  checking  for  errors,  8)  experts 
spend  proportionally  more  time  building  the  basic  problem  description  than  seeking 
solutions,  9)  experts  possess  a  larger  set  of  stored  schema,  and  10)  expert  knowledge  is 
structured,  organized,  and  inter-linked  to  a  greater  degree  (Glaser,  1985;  Glaser  &  Chi, 
1988). 

Expert  knowledge,  however,  has  been  shown  to  suffer  from  severe  biases  that 
inhibit  proper  decision-making.  Experts  as  a  group  tend  to  have  difficulty  in  responding 
to  feedback  and  shifts  in  the  nature  of  the  situation  after  a  course  of  action  has  been 
executed.  Problems  with  misleading  feedback,  limited  attention  to  delayed  feedback, 
selective  perception  of  feedback,  and  selective  influence  on  outcome  have  been 
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conclusively  shown  in  studies  (Wickens,  1993).  Novices,  due  to  their  lack  of  pre¬ 
conceived  notions  and  institutional  biases,  tend  to  react  better  to  ever  changing 
environments. 

The  results  from  these  investigations  of  skilled  behavior  all  demonstrate  that 
fundamental  differences  exist  between  expert  and  novice  decision-makers  regarding 
knowledge  structure,  pattern  perception,  performance  speed  and  accuracy,  memory 
capacity,  and  problem  perception.  Once  identified,  these  differences  cannot  only  be 
capitalized  upon,  but  if  necessary,  corrected  through  selection  methods,  training 
regiments,  or  system  modifications  (Howell,  1998). 

6.  Display  Design 

Since  it  is  the  man-machine  interface  that  most  users  are  concerned  with,  the 
question  is  now:  How  does  the  interface  designer  account  for  and  exploit  differences  in 
the  varying  levels  of  user  proficiency?  Once  the  appropriate  mental  model  of  the  user 
has  been  constructed,  how  does  a  designer  go  about  the  construction  of  a  new 
interface/di  splay?  What  principles  do  they  rely  on  to  ensure  the  user  derives  maximum 
benefit?  In  general,  a  display’s  effectiveness  is  the  functional  combination  of  five 
specific  and  distinct  attributes:  Legibility,  Readability,  Accuracy,  Compatibility,  and 
Timeliness  (Bailey,  1980). 

Legibility  simply  means  that  the  display  can  be  easily  read.  A  detailed  knowledge 
of  typefaces,  letter  sizes,  spacing,  etc.,  all  come  into  play.  If  the  display  is  illegible,  it 
will  fail  to  communicate  needed  information  to  the  user,  and  all  of  the  other 
characteristics  will  become  useless.  Legibility,  however  is  not  an  absolute  property 
(Meister,  1965).  Both  the  external  aural  and  visual  environment  affects  its  use. 
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Readability  implies  that  the  display  communicates  well,  and  the  user  perceives, 
without  effort,  the  meaning  or  purpose  of  what  is  being  presented.  This  in  turn  implies 
either  an  equal  level  of  expertise  among  users,  or  a  display  that  can  adapt  or  differentiate 
amongst  users.  Readability  is  also  achieved  in  part  by  having  the  appropriate  sequential 
order  of  presentation,  or  grouping  of  information,  to  the  user  (Ramsey  &  Atwood,  1979). 

Accuracy  is  a  double-edged  sword.  On  one  hand,  the  display  must  be  designed  to 
elicit  accurate  information  from  the  user.  Instructions  or  headings  that  are  vague, 
ambiguous,  or  misleading  may  result  in  inaccurate  entries  by  the  user.  On  the  other  hand, 
the  display  must  transmit  accurate  information  to  the  user  (Wickens,  1992).  Failure  to  do 
so  might  result  in  the  formation  of  inaccurate  hypothesis  and  the  subsequent  incorrect 
courses  of  action. 

A  display  must  be  compatible  with  the  user’s  knowledge  and  skill,  other 
concurrent  activities  being  performed,  and  again,  the  working  environment  in  which  it  is 
being  used.  Compatibility  is  perhaps  the  most  important  factor  in  both  performance  and 
acceptance  by  the  user.  Experience  has  shown  that  sometimes  systems  fail  not  for 
technical  reasons  but  for  reasons  of  user  acceptance  (Engel  &  Eruda,  1975).  The  common 
sense  remedy  to  ensure  compatibility  and  cultivate  user  acceptance  is  to  clearly  establish 
user  requirements  early  in  the  developmental  process,  and  then  ensure  that  the  new 
displays  adequately  reflects  these  requirements. 

The  final  attribute  in  display  design  concerns  the  timeliness  of  the  data  built  in 
and  presented  by  the  display.  In  simple  terms,  the  display  must  present  information 
available  to  the  user  while  it  is  still  useful.  Processing  or  compiling  (to  use  computer 
jargon)  delays  are  to  be  avoided  in  any  low-ordered  system.  Also,  a  well-designed 
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display  should  present  only  essential  information  in  such  a  form  as  to  require  only  a 
minimum  amount  of  interpretation  (Bailey,  1980).  Information,  just  for  information’s 
sake,  has  no  place  and  only  serves  to  confuse  the  operator. 

7.  Link  Analysis 

In  addition  to  the  five  factors  of  good  display  design,  two  more  issues  should  be 
addressed.  They  are  the  concepts  of  Grouping  and  Standardization.  Grouping  techniques 
are  instrumental  in  helping  to  organize  information  and  are  comprised  of  four 
fundamental  traits:  Sequence,  Frequency,  Function,  and  Importance.  Standardization 
techniques  assure  that  information  is  presented  in  compatible  sensory  modes  and  that 
they  do  not  conflict  with  any  preconceived  notions  and/or  biases  held  by  the  users. 
(Bailey,  1980). 

Link  Analysis  groupings  all  share  the  same  core  principle:  items  should  be 
grouped  together  based  on  their  relative  relationship  to  each  other  (Bailey,  1980).  This 
relationship  can  be  defined  in  terms  of  sequence  of  use  (i.e.  left  to  right  and  top  to 
bottom),  frequency  of  use  (i.e.  bivariate  normal  distributions  of  item  location  in  the 
display),  functional  allocation  (i.e.  changing  the  layout  of  the  display  based  on  operating 
mode  required  by  the  user),  and  importance  placement  (i.e.  placing  critical  items  in  the 
best  position  on  the  display  to  insure  that  they  are  not  overlooked).  Each  type  of 
grouping  lends  itself  to  different  data  collection  methods.  Sequential  relationships  are 
identified  by  using  a  process  called  procedural  flow  analysis.  Frequency  analysis  is 
accomplished  via  a  process  called  link  analysis.  Functional  and  Importance  relationships 
are  most  often  determined  by  a  combination  of  historical,  procedural,  and  subject 
operating  methods  and  modes  (Chapanis,  1959).  Each  type  of  Grouping  has  its 
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contribution  to  make,  and  it  is  the  designer’s  task  to  undertake  a  deliberate  process  of 
elimination,  weighting,  and  judgment  to  arrive  at  the  “best”  possible  design.  When 
conflicts  arise,  as  they  are  wont  to  do,  the  designer  must  give  the  highest  precedence  to 
meeting  human  performance  requirements  above  all  others. 

Standardization  itself  is  comprised  of  two  types:  standardization  of  items  on  a 
specific  display  and  standardization  of  display  format  (Bailey,  1980).  Standardization  of 
items  refers  to  the  goal  that  all  items  in  the  system  with  the  same  information  should  have 
the  same  modality.  This  modality  includes  not  only  its  name,  structure  and  physical 
appearance,  but  also  its  encoding  algorithm.  Standardization  of  display  format  is  the 
principle  in  which  the  interface  designer  attempts  to  place  items  that  appear  on  multiple 
displays  in  the  same  geographical  location  on  all  displays.  This  intentional  placement 
reduces  the  workload  on  users  by  allowing  the  user  to  capitalize  on  “location”  memories 
(Davis,  1966).  Although  standardization  is  desirable,  it  should  not  take  precedence  over 
the  grouping  principles  described  earlier. 

As  stated  before,  there  are  several  avenues  that  facilitate  proper  human  factor  data 
collection  techniques:  Procedural  Flow  Analysis,  Articulation  Testing,  Critical  Incident 
Analysis,  Time  Line  Analysis,  and  Link  Analysis  just  to  name  a  few.  Each  method  is 
used  to  address  similar,  but  very  specific  questions.  The  latter,  link  analysis  is  the 
observational  technique  for  determining  the  relative  association  among  system 
components  or  elements  (Bailey,  1980).  The  results  of  link  analysis  are  typically 
expressed  in  either  frequencies  or  probabilities.  The  textbook  application  of  link  analysis 
is  the  “office  desk  arrangement  problem”  (Chapanis,  1959).  In  this  situation,  the  office 
layout  designer  is  concerned  with  arranging  the  desks  of  several  employees  in  order  to 
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facilitate  the  greatest  communication  with  the  minimum  movement  of  employees.  Data 
is  collected  on  the  number  and  distribution  of  interactions  between  employees  and  a 
seating  arrangement  then  evolves.  However,  elementary  applications  of  link  analysis  do 
not  take  into  account  issues  of  time  partitioning,  usefulness,  or  hierarchical  associations 
among  subjects.  Therefore,  link  analysis  is  best  used  when  determining  the  layout  and 
arrangement  of  people  and  machines  in  systems,  as  well  as  the  displays  they  employ. 

B.  MILITARY  APPLICABILITY 

1.  Submarine  Approach  Methodology 

Regardless  of  a  submarine’s  operational  tasking,  successful  units  provide  the  CO 
with  quick  and  accurate  information  on  all  contacts.  The  task  of  correlating  and 
transforming  the  available  raw  data  into  the  necessary  picture  is  borne  by  the  AO 
(Clancy,  1997).  The  AO  performs  the  role  of  senior  decision-maker  and  advisor  to  the 
CO  during  an  encounter  with  a  hostile  target. 

Information  provided  to  the  AO  is,  unfortunately,  cryptic  and  uncertain.  The  AO 
will  never  be  able  to  confirm  theories  directly,  but  may  only  observe  how  closely  reality 
conforms  to  them  (Clancy,  1990).  The  AO  hunts  an  enemy  submarine  that  is  most  likely 
also  hunting  for  him.  Unlike  other  modem  forms  of  combat,  the  AO  has  no  visual 
contact  with  an  opponent.  The  medium  of  the  ocean  inhibits  visual  utilization  to  only  a 
few  feet  (Sharpe,  1998).  Rather,  those  that  operate  under  the  sea  utilize  aural  information 
exclusively. 

A  useful  analogy  of  the  AO's  task  would  be  the  hypothetical  situation  of  two 
blindfolded  prizefighters  in  a  boxing  ring.  In  the  ring,  each  fighter  seeks  the  other  and 
each  has  a  brutal  power  punch  available  to  use  if  necessary.  However,  they  are  not  alone 


22 


in  the  ring,  as  innocent  non-combatants  (referee,  trainers,  scorecard  keepers,  etc.)  and 
even  the  occasional  domestic  animal  wanders  occasionally  about.  Fortunately,  each 
knows  the  approximate  layout  of  the  ring,  i.e.  the  location  of  the  ropes,  stools,  and  other 
distinguishing  traits.  During  his  search,  each  relies  on  his  own  aural  abilities  to  identify 
other  wanderers,  decide  whether  they  might  be  the  opponent,  and  attack  if  within  range  of 
a  punch.  Still,  simply  detecting  the  enemy  does  not  imply  the  ability  to  hit  him.  First  the 
fighter  must  roughly  decide  the  distance  of  the  enemy,  using  the  level,  pitch,  and 
directivity  of  the  noise.  Once  the  enemy  range  is  identified,  the  fighter  must  approach  his 
opponent,  within  effective  striking  range,  remembering  that  the  chances  of  being  counter- 
detected  increases  each  second.  Using  similar,  but  somewhat  more  complex  methods,  the 
fighter  also  determines  the  enemy’s  moving  direction  and  speed.  Once  the  fighter  is 
satisfied  that  the  enemy’s  "solution"  has  been  determined,  a  punch  is  thrown,  and  then  the 
fighter  quickly  retreats  to  avoid  the  enemy’s  counter-punch,  which  will  assuredly  be 
thrown  in  response.  A  successful  engagement  will  find  the  fighter  safe,  his  opponent 
dispatched  by  a  wicked  right  cross,  and  no  enemy  counter-punch  contacting  the  fighter 
due  to  some  suicidal  last  effort. 

Even  the  forgoing  discussion  simplifies  the  role  of  the  AO  considerably.  Physics 
presents  difficult  challenges  in  a  real  scenario;  e.g.  sound  does  not  travel  in  straight  lines, 
rather  it  bends  and  bounces  as  it  travels  through  the  ever-changing  ocean  medium 
(Sharpe,  1998).  Received  sound  has  often  been  reflected  off  the  bottom  or  surface  of  the 
ocean,  so  the  source  lies  not  on  a  line  but  instead  a  hyperbola  opening  behind,  or  in  front 
of,  the  sensor.  The  phenomena  of  Convergence  Zones,  Ducting,  Shadow  Zones,  Arrival 
Paths,  and  Limiting  Rays,  just  to  name  a  few,  even  further  mire  the  situation.  This 
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discussion  is  only  intended  to  give  some  idea  of  the  various  tasks  and  concerns  the  AO 
faces  during  a  submerged  approach. 

2.  Previous  Research 

Due  to  the  inherently  secretive  nature  of  submarine  design  and  construction,  there 
is  a  noticeable  absence  of  unclassified  human  factor  analysis  in  the  submarine 
community.  Previous  work  has  shown  that  naval  officers  engaged  in  Anti-Submarine 
Warfare  (ASW)  and  Anti-Surface  Warfare  (ASUW)  show  a  pre-disposition  to  data 
representation  in  analogical,  or  geo-situational,  frames  of  reference  (Kirschenbaum, 
1995;  Hooper,  1994).  In  addition,  it  has  been  shown  that  while  senior  ASW  decision¬ 
makers  strive  for  optimality  in  their  endeavors,  certain  inherent  and  specific  cognitive 
limitations  and  biases  prevent  this:  they  include  1)  working  memory  limitations,  2)  a  bias 
toward  overestimation  of  enemy  submarine  maneuverability,  and  3)  a  bias  toward 
overestimation  of  sonar  accuracy  (Wohl,  1989). 

There  is  a  wealth  of  research  on  human  factor  analysis  concerning  aviation  issues, 
and  to  a  lesser  extent,  military  decision-makers  in  general.  Several  texts  (Wickens,  1992, 
Wiener  &  Nagel,  1988,  Pruitt,  1982)  give  excellent  examples  of  the  depth  and  breadth  of 
HF  aviator  work.  The  cognitive  link  between  aviators,  surface  warfare  officers,  and 
submarine  AO’s  was  established  by  Hooper  (1994),  and  is  used  as  an  anchor  for  the 
principles  applied  in  the  course  of  this  research.  It  is  in  fact  reasonable  to  assume  that,  in 
general,  military  decision-makers  all  share  the  same  cognitive  abilities  and  limitations. 

3.  Current  Research 

Unlike  the  multitude  of  research  projects  underway  in  aviation  areas,  the  only 
major  unclassified  submarine  related  human  factor  analysis  project  currently  under  works 
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is  Project  NEMO  (Kirschenbaum,  1997).  Early  work  in  project  NEMO  has  concentrated 
on  examining  the  goal  and  sub-goal  structure  of  a  small  cadre  of  senior  submarine  AO’s. 
Preliminary  results  indicate  a  very  shallow  goal  structure,  as  well  as  a  bias  toward 
analogical  schema  orientation.  Secondary  results  indicate  a  possible  limit  on  both  the 
short  term  and  long  term  working  memory  of  AO’s  that  must  be  acknowledged  and 
exploited.  Current  work  surrounds  expert  vs.  novice  issues  as  well  as  preliminary  display 
resolution  concerns.  Future  work  is  slated  for  building  a  complete  cognitive  model  of  the 
submarine  AO  (Kirschenbaum,  1997). 

C.  SUMMARY 

The  issues  presented  are  indeed  complex,  but  manageable.  Current  HIP  models, 
NDM  paradigms,  SA  measurement  methods.  Expert  vs.  Novice  distinctions,  and  their 
implication  in  modem  DSS  design  have  been  applied  to  military  settings  successfully  in  a 
variety  of  areas  (Howell,  1998).  Most  notably,  the  construction  of  TADMUS  and  most 
modem  military  aircraft  cockpits.  Only  recently  however,  have  issues  related  specifically 
to  submarine  man-machine  interface  been  freely  discussed. 

The  applicability  of  the  forgoing  discussion  to  the  AO  task  can  be  summarized  as 
follows.  By  acknowledging  and  operating  within  the  limitations  inherent  in  the  HIP 
model,  it  is  postulated  that  AO's  employ  the  NDM  paradigm  for  decision-making  in  high- 
stress  environments.  Measures  of  AO  SA,  built  in  response  to  stored  schema  fostered 
during  in  the  course  of  initial  and  periodic  training,  differentiate  between  levels  of 
proficiency  and  skill.  These  levels  are  typically  correlated  to  the  established  designation 
of  expert  vs.  novice.  After  revealing  these  expectations  of  proficiency,  data  analysis  on 
AO  informational  usage  can  be  utilized  in  the  prediction  of  future  success  and  the 
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construction  of  tools,  i.e.  displays,  that  have  the  potential  for  improving  the  AO’s  ability 
to  execute  their  primary  task,  i.e.  to  search  out  and  destroy  an  adversary.  These  same 
tools  can  also  be  used  as  a  selection  and  training  aid  in  the  course  of  an  AO's  professional 
development. 
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m.  METHODS 


A.  RESEARCH  APPROACH 

This  research  involves  the  analysis  of  a  human  performance  database  collected 
during  the  employment  of  a  Submarine  CCS  Simulator.  The  database  contains  both 
numerical  data  regarding  the  use  of  different  displays  and  information  available  in  the 
simulator  as  well  as  specific  background  data  on  the  test  subjects.  Statistical  regression 
techniques  determine  which  demographic  variables  can  be  used  to  predict  the  quality  of 
an  engagement.  Trends  in  data  usage  among  subject  groupings  are  identified  and  then 
used  as  inputs  for  the  design  of  the  next  generation  Submarine  CWS. 

B.  DATA  COLLECTION 

I.  Subjects 

Qualified  active  duty  Submarine  Warfare  officers  of  varying  experience,  as 
measured  in  years  of  Naval  service  and  sea  service  participated  in  the  study  (See  Table 
3.1).  They  represent  approximately  10%  of  the  total  available  pool  of  U.S.  Navy  AO's. 
Subjects  trials  were  conducted  in  the  Operations  buildings  of  NSB  Pearl  Harbor,  HI  and 
NSB  Bangor,  WA  during  period  of  May  18th  to  June  4th  1998.  Subjects  were  briefed  on 
the  purpose  of  Project  NEMO  and  provided  informed  consent  in  accordance  with  the 
ethical  conduct  for  subject  participation  specified  in  the  Protection  of  Human  Species, 
Secretary  of  the  Navy  (SECNAV)  instruction  3900.39b  (See  Appendix  B). 


YEARS 

MEAN 

STD  DEV 

MIN 

MAX 

NAVAL  SERVICE 

13.22 

5.10 

5.00 

23.00 

SEA  SERVICE 

6.26 

2.92 

1.50 

14.00 

Table  3.1  Subject  Experience. 


2.  Apparatus 

A  Macintosh  PowerBook  2400c/180  (603E)  equipped  with  an  11.1”  active  matrix 
screen  presents  the  stimulus.  Subject  interaction  with  the  simulation,  to  include 
vocalization,  is  recorded  via  a  PC-to-VCR  s-port  adapter,  a  Sony  HI-8  TR9  camcorder, 
and  a  lapel  microphone.  Subjects  are  seated  in  front  of  the  computer  and  are  allowed  free 
movement  during  the  testing.  Figure  3.1  shows  the  equipment  configuration.  To  avoid 
any  possible  distraction,  the  entire  apparatus,  with  the  exception  of  the  laptop  and  mouse, 
is  covered  with  an  opaque  black  cloth. 


Figure  3.1  Experiment  Equipment  Configuration. 


3.  Stimulus 

The  simulation  mimics  the  current  tactical  displays  onboard  a  generic  U.S. 
submarine.  Ten  separate  displays  are  presented;  OwnShip-Control  (OSC),  Sound 
Velocity  Profile  (SVP),  Line  of  Sight  (LOS),  Geo-Plot  (GEO),  Time-Bearing  (TB), 
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Time-Frequency  (TF),  Spherical  Array  Passive  BroadBand  (SAPBB),  Towed  Array 
Passive  BroadBand  (TAPBB),  Towed  Array  Passive  NarrowBand  (TAPNB),  and 
Primary  Mate  (MATE).  The  simulation  contains  one  practice  and  four  actual  test 
scenarios.  The  four  scenarios  are  representative  of  the  expected  tactical  encounters  for 
which  U.S.  AO’s  train.  However,  the  difficulty  of  the  scenarios  is  purposely  skewed 
toward  the  easier  side  in  order  to  facilitate  increased  user  interaction.  The  simulation 
language  is  Mac  Common  LISP. 

4.  Instrument 

The  instrument  consists  of  two  elements.  The  first  is  the  layering  of  a  MSExcel 
OLE  compliant  recorder  underneath  the  stimulus.  Informational  use  by  the  subject  is 
recorded  by  what  is  referred  to  as  a  “poor  man’s  eyetracker.”  This  instrument  works  in 
the  following  manner:  1)  Each  display  is  accessed  individually  by  use  of  a  vertical  task 
bar  on  the  side  of  the  “master”  display.  2)  All  information  on  the  displays  is  normally 
covered  by  the  use  of  “black  boxes”  generated  by  simulation.  To  view  a  piece  of 
information  on  a  specific  display,  the  subject  places  the  computer-generated  cursor  over 
the  display  area  to  be  examined  and  depresses  a  button.  Once  the  button  is  depressed,  the 
black  box  for  that  specific  piece  of  information  is  removed  and  the  subject  is  allowed  to 
view  the  requested  information.  3)  The  information  is  displayed  only  as  long  as  the 
cursor  is  held  steady.  Layering  the  recorder  under  the  stimulus  allows  for  the  archiving 
of  the  events  that  occur  during  the  course  of  the  experiment.  The  archived  data  for  each 
subject  trial  is  then  stored  for  future  examination.  The  second  instrument  is  a  small 
survey  questionnaire  designed  to  measure  subject  satisfaction  as  well  as  the  perceived 
difficulty  and  fidelity  of  the  stimulus.  The  survey  also  measures  the  subject  utilization  of 
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actual  CCS  displays  and  provides  an  avenue  for  subject  input  to  the  CWS  design.  (See 
Appendix  D) 

5.  Procedure 

The  protocol  guides  the  interviewer  through  the  process  of  subject  identification, 
indoctrination,  testing,  classification,  and  subjective  evaluation.  Subjects  are  identified 
by  the  interviewer  via  a  sequenced  3-digit  number  and  that  number  is  used  throughout  the 
remainder  of  the  testing  process.  Subject  indoctrination  consists  of  two  phases.  First, 
subjects  view  a  short,  60  second,  computer  video  on  how  to  play  a  tic-tac-toe  game  in 
which  the  computer  verbalizes  it’s  thoughts  and  actions.  Next,  they  are  allowed  to  play 
the  same  tic-tac-toe  game  and  are  instructed  on  how  to  verbalize  their  own  thoughts  and 
actions.  They  are  then  instructed  to  continue  verbalization  throughout  the  remainder  of 
the  testing  process.  Next,  subjects  are  taken  on  a  "guided  tour"  of  the  simulation  by  the 
interviewer  and  instructed  on  how  the  simulation  works.  A  short,  15-minute,  free-play 
period  then  ensures  that  the  subject  is  "comfortable"  with  the  actual  operation  of  the 
tactical  simulation.  The  testing  phase  commences  with  each  subject  receiving  the  same 
mission  statement  shown  in  Figure  3.2. 

"The  world  situation  is  that  a  war  has  broken  out  between  Russia  and  Ukraine 
over  control  of  the  former  Soviet  Navy.  Both  nations  are  trying  to  get  the  US  involved 
and  have  been  attacking  U.S.  shipping  with  submarines  and  then  blaming  each  other. 

You  are  patrolling  in  the  eastern  Caribbean  Sea.  Your  mission  is  to  protect  U.S.  shipping 
lanes.  Your  orders  are  to  search  and  destroy  any  enemy  submarines  in  the  vicinity  of 
U.S.  shipping.  The  month  is  April  and  the  sea  state  is  2." 

Figure  3.2  Mission  Statement. 
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Each  subject  attempts  to  complete  two  of  the  four  available  scenarios.  Each 
scenario  starts  with  a  status  report  similar  to  that  shown  in  Figure  3.3. 

STATUS  REPORT 

"You  hold  one  merchant  bearing  300  degrees,  but  no  subsurface  contacts.  The 
broadband  range  of  the  day  for  a  subsurface  contact  is  3-6  thousand  yards.  Intelligence 
has  indicated  an  Alfa-type  Formerly  Soviet  Submarine  is  patrolling  in  the  area.  There  are 
no  friendly  submarines  in  your  patrol  area." 

Figure  3.3  Status  Report. 

The  four  initial  contact  scenarios  differ  only  in  initial  contact  bearing,  course  and 
speed.  The  number  of  contacts  and  intelligence  report  remains  the  same  throughout  the 
experiment  (See  Appendix  C). 

At  the  conclusion  of  the  testing  phase,  the  subjects  provide  background  data  on 
themselves  and  complete  a  subjective  evaluation  of  the  simulation.  The  final  step  is  the 
subjective  design  phase.  At  this  time  the  subject  is  asked  to  produce  a  sketch  of  what  the 
subject  "wants"  on  the  next  generation  of  Submarine  CWS  if  they  were  the  one  in  charge. 
Discussion  is  encouraged  during  this  time  with  the  tester  as  to  facilitate  creativity  and 
innovation  (See  Appendix  B  for  the  full  protocol). 

6.  Experimental  Design 

A  3  x  2  x  4  modified  Latin  square  design  is  used  in  the  experimental  protocol  (See 
Appendix  C).  The  three  refers  to  the  3  levels  of  operator  proficiency  (Expert,  Mid, 
Novice).  The  two  refers  to  the  run  order  (First  vs.  Second)  of  the  two  scenarios 
presented.  And  the  four  refers  to  the  designation  of  the  scenario  (Cl,  C2,  Zl,  Z2) 
presented.  The  modified  design  was  chosen  because  the  distribution  of  the  subject 
proficiency  was  not  known  in  advance  of  the  data  collection.  This  structure  allows  for  a 
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balanced  design  across  the  levels  of  subject  proficiency,  presentation  order,  and  target 
scenario. 

C.  DATA  ANALYSIS 

1.  Data  Generation 

Five  events  are  recorded  by  the  first  element  of  the  instrument:  the  specific 
display  viewed,  the  specific  information  viewed  on  the  display,  the  total  time  spent 
viewing  the  information,  any  inputs  the  subject  made  to  the  simulation,  and  the  time  at 
which  the  viewing  of  or  input  to  the  specific  display  occurred.  Each  event  is  stored  as  a 
separate  line  of  ASCII  data  and  the  scenario  in  total  is  stored  as  an  individual  computer 
file.  The  instrument's  second  element,  the  survey  questionnaire,  consists  of  four  closed- 
end  and  three  open-ended  questions.  The  four  closed  questions  measure  scenario  realism, 
scenario  difficulty,  display  effectiveness  and  display  utility.  Three  of  the  closed 
questions  use  a  5-choice  Likert  scale  while  the  last  uses  a  10-choice  ordinal  ranking 
layout.  The  three  open  questions  measure  overall  subject  response  to  the  experiment, 
subjective  desires  for  CWS  attributes,  and  general  feedback.  The  questionnaires  are 
completed  by  the  subjects  in  the  presence  of  the  interviewer. 

2.  Data  Tabulation 

The  computer  generated  ASCII  files  are  first  individually  converted  into  an  MS 
Excel  spreadsheet  and  a  Visual  Basic  (VB)  Macro  is  applied  (See  Appendix  F).  The 
individual  trial  spreadsheets  are  combined  together  with  the  survey  responses  and  then  all 
of  the  data  is  merged  to  form  a  "master"  spreadsheet  which  is  organized  by  both  subject 
and  trial  number  to  facilitate  statistical  analysis  by  demographical  as  well  as  scenario 
distinctions.  The  master  spreadsheet  is  available  in  both  MS  Excel  and  SPLUS  formats. 
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3.  Variable  Description 

The  predictor  variables  are  of  two  categories:  one,  demographic  variables 
describing  the  subjects,  and  two,  CCS  display  usage  patterns  and  subject  survey  results. 
Subject  demographic  variables  include  six  categorical  variables:  Position  (POS;  1 -Expert, 
0-Novice),  Ship  Type  (TYPE;  1-SSN,  0-SSBN),  Geographical  Location  (LOC;  1-Pearl 
Harbor,  O-Bangor),  Duty  Type  (ROT;  1-Sea  Duty,  0-Shore  Duty),  Level  of  Education 
(EDU;  1-Masters/Ph.D.,  0-BS/BA),  and  Scenario  Completed  (SCEN;  1-C1/C2,  0-Z1/Z2). 
The  display  usage  variables  are  functions  of  four  basic  variables:  the  number  of  display 
accesses,  the  total  time  of  display  use,  the  average  time  of  display  use  per  access,  and  the 
standard  deviation  of  time  of  display  use  per  access.  These  four  variables  combined  with 
the  survey  results  produce  92  distinct  display  usage  variables.  The  demographic  and 
display  usage  variables  are  extracted  from  the  original  ASCII  files  by  use  of  the 
previously  mentioned  VB  Macro. 

The  response  variables  take  the  form  of  two  continuous  objective  and  one  binary 
subjective  MOE's.  The  two  continuous  objective  measures  are  solution  accuracy  at  time 
of  fire  (TOF)  and  weapon  to  target  closest  point  of  approach  (CPA).  To  obtain  an 
objective  measure  of  accuracy  at  TOF,  the  absolute  distance  between  the  actual  target 
location,  as  recorded  by  the  simulation,  and  the  perceived  target  location,  recorded  as  the 
firing  solution  of  the  subject  just  prior  to  TOF,  is  computed.  To  standardize  the  geometry 
across  all  subjects,  a  2-D  grid  is  superimposed  on  the  engagement  area  with  the  AO  at  the 
center  (0,0)  and  the  targets,  actual  and  perceived,  identified  by  their  relative  positions  on 
the  grid.  This  error,  termed  Composite  Error  (CE)  is  given  in  Equation  1. 
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CE  =  ((Xa-Xp)2+( Y a- Y p)2)1/2  [yds]  Eqn.  (1) 

where, 

(Xa,  Ya)  :  Actual  target  position  [yds]  on  the  2-D  grid 
(Xp,  Yp)  :  Perceived  target  position  [yds]  on  the  2-D  grid 
The  calculation  of  weapon  to  target  CPA  is  determined  via  a  two-stage  process.  First,  the 
final  firing  solution  is  extracted  from  the  simulation  and  the  appropriate  weapon 
kinematic  parameters  are  generated.  The  weapon  is  assumed  to  be  a  standard  U.S.  Navy 
MK-48  mod  IV  Heavyweight  Torpedo  (See  Appendix  E).  From  there,  a  GAMS 
simulation  determines  the  actual  torpedo  to  target  CPA  (See  Appendix  G)  in  terms  of  the 
CPA  range  (Rcpa)  and  CPA  time  (Tcpa). 

To  obtain  the  subjective  assessment  of  subject  performance,  three  factors  are 
examined:  torpedo  to  target  CPA,  AO  exposure,  and  time  of  engagement.  These  three 
factors  answer  the  following  questions.  First,  would  the  final  solution  have  resulted  in  a 
HIT?  Second,  did  the  AO  minimize  own-ship’s  EXPOSURE  to  counter-detection  by  the 
opposing  submarine?  And  third,  did  the  AO,  upon  achieving  a  “good”  firing  solution, 
realize  that  the  engagement  could  end  at  that  time  and  thus  minimize  the  TIME  of  the 
engagement?  It  is  assumed  that  if  all  three  factors  are  within  the  acceptable  ranges,  the 

engagement  as  a  whole  would  be  successful,  i.e.  the  scenario  resulted  in  the  destruction 

| 

of  the  enemy  and  the  survival  of  own  ship.  If  any  one  is  outside  the  prescribed  range, 
then  the  engagement  as  a  whole  would  be  considered  unsuccessful,  i.e.,  the  scenario 
resulted  in  the  survival  of  the  enemy  and/or  the  destruction  of  own  ship. 

The  notion  of  the  AO  getting  a  weapon  hit  is  the  first  factor  examined.  A  HIT  is 
defined  to  have  occurred  if  the  final  solution  generated  weapon  parameters  would  have 
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resulted  in  an  impact  with  the  enemy.  This  factor,  classified  as  a  1  for  an  impact  and  a  0 
otherwise,  is  determined  via  a  two-stage  process.  First,  the  final  firing  solution  entered 
by  the  subject  is  extracted  from  the  simulation.  And  second,  as  with  the  determination  of 
CE,  the  appropriate  CPA  parameters  are  generated.  If  the  CPA  range  is  less  than  acoustic 
acquisition  range  for  the  torpedo  (2000  yds)  and  the  CPA  time  is  within  the  weapons  run 
time  (3500  sec),  the  weapon  is  assumed  to  impact  the  target. 

AO  exposure  is  the  next  factor  examined.  EXPOSURE,  in  this  case,  is  defined  as 
the  minimum  range  experienced  by  the  AO  during  the  course  of  the  engagement.  Two 
factors  are  used  in  determining  a  cutoff  threshold.  First,  it  is  assumed  that  the  enemy  is 
pursuing  the  AO  and  while  it  is  assumed  that  the  AO  has  an  acoustical  advantage,  the 
chance  of  counter-detection  increases  as  exposure  decreases.  Second,  to  coin  a  movie 
phrase,  "the  problem  with  submarines  is  that  they  do  not  turn  on  a  dime"  (Clancy,  1990). 
The  point  being  that,  as  two  submarines  get  close  to  one  another,  the  risk  of  collision  is 
just  as  important  as  the  risk  of  counter-detection.  For  these  two  reasons,  a  lower  limit  of 
2000  yards  is  placed  on  EXPOSURE.  If  the  AO  to  enemy  range  is  greater  than  2000 
yards  for  the  duration  of  the  problem,  a  1  is  recorded  and  if  not,  a  0. 

The  time  required  to  get  a  solution  is  the  final  factor  examined  in  determining 
overall  solution  quality.  While  longer  engagements  can  lead  to  more  accurate  solutions, 
they  also  increase  the  danger  to  the  AO.  Remember  that  while  the  AO  is  pursuing  the 
enemy,  the  enemy  is  also  pursuing  the  AO.  Hence  the  danger  of  counter-detection  by  the 
enemy,  and  resulting  counter-fire,  increases  as  the  AO  spends  more  time  in  the  vicinity  of 
the  enemy.  A  perfect  solution  obtained  after  the  enemy  has  disengaged  or  while  getting 
shot  is  of  no  value.  Therefore,  to  account  for  this,  a  TIME  constraint  of  1500  seconds  is 
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used  to  identify  the  point  at  which  counter-detection  and  eventual  counter-fire  might 
occur  by  the  target.  If  the  AO  initiated  a  firing  prior  to  this  time,  a  1  is  recorded  and  if 
not,  a  0. 

These  three  factors:  HIT,  EXPOSURE,  and  TIME  are  combined  to  determine  if 
the  engagement  is  a  success.  If  all  three  factors  are  a  1,  the  engagement  is  a  success  and 
a  1  is  recorded  for  the  variable  SF  (success/failure).  If  any  of  the  factors  is  a  0,  then  the 
engagement  as  a  whole  is  a  failure  and  a  0  is  recorded  for  SF. 

4.  Statistical  Analysis 

To  analyze  and  compare  the  performance  of  the  subjects,  a  two  pronged  approach 
is  used.  First,  performance  is  studied  as  a  function  of  the  subject  demographic  variables. 
For  this,  the  two  MOE’s,  CE  and  SF,  are  used.  Classical  logistical  regression  (Hamilton, 
1992)  and  CART  analysis  (SPLUS,  1996)  compromise  the  bulk  of  the  data  analysis 
techniques  employed.  The  second  prong  will  explore  differences  in  display  usage  and 
solution  generation  strategies  of  the  subjects  as  a  function  of  the  demographic  variables. 
This  approach  is  shown  in  Figure  3.4. 


The  display  design  process  will  utilize  a  three  pronged  approach.  First,  the 
subject  survey  evaluation  of  the  effectiveness  and  usefulness  of  the  current  CCS  displays 
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(See  Appendix  D)  as  measured  by  the  mean  response  will  be  extracted.  Second,  link 
analysis  techniques  will  extract  the  display  usage  and  solution  generation  patterns  of  the 
"best"  group  of  AO's  as  distinguished  by  distributional  and  categorical  relationships 
previously  identified.  And  third,  the  top  ten  salient  issues  generated  by  subject  feedback 
when  asked  of  their  desires  for  the  next  generation  of  CWS  (See  Appendix  D)  will  be 
used  as  the  final  input.  These  three  inputs  will  then  be  merged  into  a  comprehensive 
layout  and  set  of  display  attributes  for  the  final  CWS.  This  design  methodology  is  shown 
in  Figure  3.5. 
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IV.  RESULTS 


The  analysis  conducted  in  this  research  is  primarily  exploratory  in  nature.  Over 
100  hours  of  experimental  data  is  utilized  in  the  creation  of  a  subject  performance 
database.  From  the  database,  the  variables  most  indicative  of  subject  performance  are 
extracted  and  used  as  the  basis  for  analysis.  Any  subject  modeling  or  statistical  analysis 
results  obtained  will  need  to  be  verified  by  an  independent  study  prior  to  drawing  any 
final  conclusions. 

A.  TRIAL  RESULTS 

The  subject  pool  for  the  experiment  consists  of  49  volunteers.  Of  these,  36 
subjects  correctly  finished  the  testing  protocol  resulting  in  65  completed  trials.  Even  with 
the  exclusion  of  26  percent  of  the  experiment  participants,  the  use  of  the  modified  Latin 
square  design  for  the  remaining  65  trials  succeeded  in  partitioning  the  subject  pool  fairly 
evenly  according  to  the  6  demographic  variables.  This  is  shown  in  Table  4.1. 


Demographic  Variable 

Variable  =  1 

Variable  =  0 

n  (%) 

n  (%) 

POS  (1  -  Expert,  0  -Novice) 

27  (42) 

38  (58) 

TYPE  (1  -  SSN,  0  -SSBN) 

33  (51) 

32  (49) 

LOC  (1-  Pearl  Harbor,  0  -  Bangor) 

28  (43) 

ROT  (1  -  Sea  Duty,  0  -  Shore  Duty) 

50  (77) 

■dgm 

EDU  (1  -  MS/MA/PhD,  0  -  BS/BA) 

34  (52) 

31  (48) 

SCEN  (1  -  C1/C2,  0  -  ZI/Z2) 

33  (51) 

32  (49) 

Table  4. 1  Subject  Pool  Partitioning  by  Demographic  Variables. 

Subject  responses  to  the  issues  of  simulation  realism  and  scenario  difficulty  are 
first  examined  to  verify  the  original  mandates  of  simulation  fidelity  and  ease  of  use.  As 
is  shown  in  Figure  4.1  and  4.2,  the  frequency  histograms  of  Realism  and  Difficulty  are 
right-skewed,  with  a  median  of  four  and  a  mean  of  3.  With  the  scales  used,  i.e.  1-Very 
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Difficult/Unrealistic  and  5-Very  Easy/Realistic,  the  results  indicates  that,  as  a  group,  the 
subjects  rated  the  simulation  as  realistic  and  fairly  easy  to  use. 


1  2  3  4  5 


Difficulty 

Figure  4. 1  Histogram  of  Scenario  Difficulty  (1-Very  Difficult,  5-Very  Easy). 


1  2  3  4  5 


Realism 

Figure  4.2  Histogram  of  Scenario  Realism  (1-Very  Unrealistic,  5-Very  Realistic). 

B.  MOE  RESULTS 

When  analyzing  and  distinguishing  the  performance  of  the  subjects,  the  earlier 
defined  two  pronged  approach  is  used.  The  distributions  of  the  MOE's,  CE  and  SF,  along 
with  CPA,  EXPOSURE,  and  TIME,  are  examined  first  in  an  attempt  to  understand  the 
underlying  relationships  between  the  MOE’s.  Excluding  TIME,  the  distributions  indicate 
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that  the  subjects  performed  within  the  ranges  of  values  expected.  The  distribution  of 
TIME,  however,  did  not  reveal  the  expected  bias  toward  shorter  engagement  times. 

The  distribution  of  CE  is  shown  in  Figure  4.3.  As  can  be  seen,  the  frequency 
distribution  of  CE  is  heavily  left-skewed  with  most  subjects  having  a  low  CE  with  a 
median  of  1291  yds  and  a  mean  of  1386  yds.  This  indicates  that  most  subjects  arrived  at 
a  reasonably  accurate  solution  at  TOF. 


CE(yds) 

Figure  4.3  Frequency  Histogram  of  CE. 

The  distribution  of  the  weapon  to  target  CPA  parameters,  TCpa  and  Rcpa,  are 
shown  in  Figure  4.4  and  4.5.  As  can  be  seen,  the  frequency  distributions  are  left-skewed 
(median(TcpA)  =  127  secs,  mean(TcpA) =  146  secs;  median(RcPA)  =  248  yds,  mean  (Rcpa) 
=  391  yds)  indicating  most  subjects  made  limited  use  of  the  torpedo’s  capabilities  at 
TOF.  Rather  it  seems  that  the  subjects  preferred  to  rely  on  a  more  accurate  solution  (i.e. 
a  low  CE)  at  TOF. 
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34  71  108  145  182  219  256  293  330  367 

Tcpa  (sec) 

Figure  4.4  Frequency  Histogram  of  Tcpa- 
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111  292  473  654  835  1016  1197  1378  1559  1740 

Rcpa  (yds) 

Figure  4.5  Frequency  Histogram  ofRcpA- 


The  distribution  of  exposure  is  shown  in  Figure  4.6.  As  can  be  seen,  the 
frequency  distribution  of  exposure  is  lightly  right-skewed  with  a  median  of 4000  yds  and 
a  mean  of  3903  yds.  This  indicates  that  most  subjects  had  a  high  exposure.  This  is 
consistent  with  the  expected  results  in  that  most  subjects  should  have  made  a  concerted 
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effort  not  to  get  too  close  to  their  adversary  and  therefore  risk  counter-detection  and/or 
collision. 


Exposure  (yds) 

Figure  4.6  Frequency  Histogram  of  Exposure. 

The  distribution  of  the  Time  is  shown  in  Figure  4.7.  As  can  be  seen,  the 
frequency  distribution  of  time  is  fairly  evenly  distributed,  with  approximately  equal 
median  and  mean,  indicating  a  full  spread  of  engagement  times  among  subjects  and  no 
significant  biasing  toward  either  a  long  or  short  engagement  duration.  A  Kolmorgorov- 
Smirnov  Goodness-of-fit  (KS)  test  for  uniformity  fails  to  reject  a  uniform  distribution 
with  a  p-value  of  0.76.  This  result  is  surprising  considering  the  instruction  set  given  to 
the  participants.  It  was  thought  that  the  subjects  as  a  whole  would  have  shot  much  more 
quickly.  Coupled  with  the  earlier  lack  of  reliance  on  the  torpedo’s  capabilities,  as  shown 
in  the  distribution  of  the  weapon  to  target  CPA  parameters,  the  subjects  seem  to  show  a 
bias  toward  a  more  conservative  approach  methodology. 
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Figure  4.7  Frequency  Histogram  of  Time. 


An  examination  of  the  probability  cross-tabulations  between  the  binary  variables 
of  TIME,  EXPOSURE,  and  HIT,  as  shown  in  Table  4.2,  reveals  noticeable  dependence. 
A  x2  test  for  independence  indicates  that  the  three  variables  are  indeed  dependant 
(X24=10.63,  p-value=.03).  Therefore,  subjects  that  succeed  in  one  measure  are  more 
likely  to  succeed  in  another  and  visa  versa. 


HIT=0 

HIT=1 

TIMEYEXPO  SURE 

1 

TIMEVEXPOSURE 

0 

1 

0 

0 

1 

1 

0.06 

0.80 

Table  4.2  HIT  vs.  EXPOSURE  vs.  TIME  Probability  Cross-Tabulations. 

As  outlined  earlier,  the  three  binary  factors  of  HIT,  EXPOSURE,  and  TIME,  are 
multiplied  to  determine  if  the  engagement  is  a  success.  Recall  that  if  all  three  factors  are 
classified  with  a  1,  the  engagement  is  a  success  and  a  1  is  recorded  for  the  variable  SF.  If 
any  are  classified  with  a  0,  then  the  engagement  as  a  whole  is  a  failure  and  a  0  is  recorded 
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for  SF.  Using  that  criteria,  the  subject  population,  as  a  group,  achieved  a  77%  success 
rate  (n  =  50)  and  a  23%  failure  rate  (n  =  15). 

C.  SUBJECT  MODELS 

The  next  step  is  to  determine  which  models,  if  any,  can  be  used  to  adequately 
describe  subject  performance.  Due  to  the  availability  of  both  continuous  and  binary 
response  variables  (CE  and  SF),  that  capture  two  different  aspects  of  subject 
performance,  and  the  six  demographical  predictor  variables,  two  different  models  are 
developed. 

1.  CE  Models 

First,  plots  of  CE  and  Time  vs.  all  seven  (POS,  TYPE,  LOC,  EDU,  ROT,  SCEN, 
and  SF)  demographic  variables  are  examined  for  differences  in  the  mean  value  of  CE  and 
Time.  These  are  shown  in  Figures  4.8  and  4.9.  As  can  be  seen,  the  differences  in  mean 
Time  are  considerably  less  than  the  differences  in  mean  CE.  This  indicates  that  except 
for  the  variable  SF,  there  appears  to  be  no  significant  interaction  between  Time  and  the 
six  remaining  subject  demographic  variables.  Alternatively,  there  are  noticeable 
interactions  between  CE  and  the  demographic  variables  of  SF,  LOC,  ROT,  and  TYPE. 
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Figure  4.8  Mean  CE  (yds)  by  Demographic  Variables. 
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Figure  4.9  Mean  Time  (sec)  by  Demographic  Variables. 


A  Welsh's  modified  two  sample  T-test  is  run  on  the  differences  in  mean  CE  and 


mean  Time  with  respect  to  each  of  the  seven  demographic  variables  to  determine  if  the 
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results  were  indeed  significant1.  As  was  the  postulated  earlier,  the  factors  of  SF,  TYPE, 
LOC,  EDU  and  ROT  show  significant  effects  for  CE,  while  only  SF  shows  significant 
effects  for  Time.  We  note  that  with  the  number  of  two  sample  tests  performed,  the  p- 
value  becomes  a  descriptive  statistic.  Nevertheless,  the  p-values  still  indicate  that  there  is 
"strong"  evidence  that  there  is  a  difference  in  the  expected  MOE's  between  the  two 
groups.  However,  the  exact  level  of  significance  of  the  test  is  somewhat  more  difficult  to 
quantify.  The  outcome  in  the  form  of  the  mean  values  and  p-values  is  shown  in  Table 
4.3. 


Response 

Variable 

o 

l 

1 

£ 

Variables 

Mean  Time 

SF 

1557.91 

947.46 

<0.01 

POS 

1066.03 

1046.01 

0.82 

TYPE 

1084.55 

1032.35 

0.55 

LOC 

1073.93 

1038.69 

0.69 

EDU 

1023.36 

1093.53 

0.42 

SCEN 

1061.39 

1055.12 

0.94 

ROT 

1045.32 

1063.00 

0.86 

Mean  CE 

SF 

1617.03 

1334.33 

0.03 

1458.96 

0.62 

1263.46 

0.02 

LOC 

1529.99 

1201.67 

0.01 

1280.45 

1491.01 

0.06 

SCEN 

1371.44 

1401.86 

0.85 

ROT 

1597.68 

1312.19 

0.02 

Table  4.3  Descriptive  P-values  for  Differences  in  Mean  CE  and  Mean  Time. 

Next,  speed  (Time)  vs.  accuracy  (CE)  is  fit  as  a  function  of  the  seven 
demographic  variables.  This  is  accomplished  by  linearly  regressing  CE  on  Time  with 
each  of  the  seven  demographic  variables  considered  separately.  In  each  case,  the 
interaction  of  the  demographic  variables  with  the  factor  Time  is  also  included.  Figures 
4.10  through  4.16  show  the  results  of  these  multiple  regressions.  As  can  be  seen,  the 
variables  of  SF  (F2;63=4.03,  p-value<01),  LOC  (F2>63=3.09,  p-value<01),  TYPE 


1  For  the  purposes  of  this  research,  significance  is  defined  as  p-value<=.10. 
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(F2,63=3.06,  p-value<01),  EDU  (F2,63=2.76,  p-value=.01)  and  ROT  (F 2,63=2.42 ,  p- 
value=.01)  all  show  significant  effects.  The  variables  of  SCEN  (F2,63=-37,  p-value=.34) 
and  POS  (F2,63=.35,  p-value=.35)  were  negligible.  These  first  results,  particularly  those 
from  the  POS  regression,  contradicts  the  expected  notion  of  the  distinction  between 
experts  and  novices,  and  raises  the  issue  of  a  possible  reevaluation  in  the  use  of  military 
rank  as  the  primary  designator  of  expert  and  novice.  In  addition,  the  significance  of  the 
ROT  factor  indicates  that  AO  skills  are  fragile  in  that  those  subjects  that  utilize  their 
skills  on  a  frequent  basis  maintain  a  better  mastery  of  the  subtleties  of  the  task,  while 
those  that  are  otherwise  engaged  experience  a  degradation  in  their  abilities. 


Figure  4.10  Speed  (Time)  vs.  Accuracy  (CE)  Regression  Conditioned  on  SF. 
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Figure  4.11  Speed  (Time)  vs.  Accuracy  (CE)  Regression  Conditioned  on  EDU. 
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Figure  4. 12  Speed  (Time)  vs.  Accuracy  (CE)  Regression  Conditioned  on  LOC. 


cefyds): 


Additionally,  when  the  sign  of  the  slope  of  the  regression  line  was  examined  in 
under  all  possible  combinations  of  subjects  (27),  only  one  combination  (POS=0,  LOC=l, 
TYPE=1,  EDU=1,  ROT=l;  n=6)  showed  a  negative  value.  A  binomial  sign  test  applied 
to  the  null  hypothesis  that  the  probability  of  a  negative  slope  is  1/2  against  the  alternative 
hypothesis  that  the  probability  is  not  1/2,  finds  significance  in  that  in  no  case  does  a 
longer  engagement  lead  to  a  more  accurate  solution.  Therefore,  it  appears  that  an  AO's 
best  solution  is  the  first,  and  any  further  refinement  may  be  of  little  value. 

Finally,  a  full  model  fitting  CE  vs.  all  seven  demographical  variables  and  their 
interaction  with  Time  is  examined.  Using  Mallow’s  Cp,  a  stepwise  linear  regression  is 
conducted.  The  variables  of  SF,  Time,  and  SF  x  Time  comprise  the  best-fit  model  with  a 
multiple  correlation  coefficient  of  0.31.  Table  4.4  shows  the  estimated  coefficients  of  the 
predictor  variables  and  the  associated  p-values  remaining  in  the  “best  fit”  model.  These 
results  loosely  correspond  with  the  earlier  results  obtained  from  the  examination  of  mean 
CE  and  the  speed  vs.  accuracy  trials.  Examination  of  the  full  model  reveals  that  the 
highest  correlation  possible  between  CE  and  the  fitted  CE  occurs  with  value  of  0.45. 
Therefore,  with  the  three  factors  in  the  best-fit  model,  69  percent  of  the  total  possible 
model  correlation  is  attained. 


Variable 

Coefficient 

SE 

p-Value 

SF 

1036.86 

451.11 

0.02 

Time 

1.06 

0.17 

<0.01 

SF  x  Time 

-0.75 

0.49 

0.10 

Table  4.4  “Best  Fit”  Linear  Regression  Model  Coefficients  and  p-Values. 
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2.  SF  Models 


The  second  model  examined  is  a  logistic  model  with  factor  SF  as  the  binary 
response  and  the  six  subject  demographic  variables  (EDU,  LOC,  POS,  ROT,  SCEN,  and 
TYPE)  as  the  binary  predictors.  Construction  of  the  model  follows  two  separate 
methodologies. 

First,  the  model  is  fit  with  the  full  data  set.  This  allows  for  the  estimation  of  the 
expected  probability  of  success  for  each  subject  based  on  the  particular  combination  of 
predictor  variables.  Since  these  probabilities  range  from  0  to  1  and  the  actual  response 
variable  was  binary,  all  fitted  probabilities  greater  than  0.50  are  assigned  a  value  of  1  and 
all  those  probabilities  less  than  0.50  are  assigned  a  value  of  0.  The  model  is  tested  by 
examining  the  correlation  coefficient  between  the  actual  and  predicted  success  vectors. 
As  shown  in  Table  4.5,  the  factors  of  TYPE,  LOC,  and  POS  have  the  greatest  impact 
with  a  model  correlation  of  0. 8 1 . 

Second,  the  data  was  randomly  split  into  two  separate  sets  each  of  equal  size.  The 
model  was  fit  to  half  of  the  data  set  and  then  applied  to  the  other  half  in  the  prediction  of 
success.  This  was  repeated  10  times  for  standardization.  As  shown  in  Table  4.5,  the 
average  correlation  between  actual  and  predicted  success  for  the  10  repetitions  is  .76.  As 
with  the  full  model,  the  factors  of  TYPE,  LOC  and  POS  have  the  greatest  influence.  In 
both  the  full  and  ten  random  models,  errors  in  the  prediction  of  success  are  predominately 
due  to  an  overestimation  of  a  subjects  abilities  rather  than  underestimation. 
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VARIABLE 

BETA 

FULL  MODEL 

10  RANDOM  MODELS 

POS 

PI 

0.87 

1.32 

TYPE 

P2 

5.71 

5.13 

LOC 

P3 

-5.57 

-4.91 

EDU 

P4 

-0.04 

0.06 

SCEN 

P5 

0.72 

0.81 

ROT 

P6 

0.68 

0.71 

Correlation  coefficient 

r 

0.81 

0.76 

Table  4.5  Regression  and  Correlation  Coefficients  for  the  Full  and  Random  SF  Models. 

In  an  attempt  to  visualize  the  results  of  the  logistic  regression  model  of  subject 
performance,  CART  techniques  (SPLUS,  1997)  are  utilized.  Figure  4.17  shows  the 
classification  tree  for  the  subject  performance  model.  The  estimated  probability  of 
success  (Ps)  for  each  leaf  is  at  the  terminal  node.  Due  to  the  nature  of  the  data,  i.e.  binary 
values  independent  of  each  other,  the  CART  tree  is  ideal  for  quickly  identifying  a  subject 
by  their  demographic  variables  and  their  associated  probability  of  success.  For  example, 
for  an  AO  from  Bangor  (LOC=0),  with  a  BS  degree  (EDU=0),  attempting  scenarios 
C1/C2  (SCEN=0),  we  follow  the  tree  down  and  to  the  left  to  get  an  estimated  probability 
of  success  of  0.75.  As  with  earlier  models,  the  factors  of  LOC,  EDU,  and  ROT  are 
significant  while  yet  again,  the  factor  of  POS  is  negligible.  The  misclassification  rate  for 
the  CART  model  was  .22.  When  compared  to  the  correlation  coefficients  arrived  at  with 
the  logistical  model  (r=0.81,  r=0.76),  we  find  that  the  CART  analysis  arrives  at  similar 
conclusions. 
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Figure  4.17  CART  Estimated  Probability  of  AO  Success  based  on  Demographic 

Variables. 

The  second  approach  examines  the  relationships  between  the  92  dependent 
variables  of  display  usage  and  solution  generation,  and  the  7  (SF,  POS,  LOC,  EDU,  ROT, 
TYPE,  and  SCEN)  independent  demographic  variables.  Because  the  measurement  scales 
of  the  dependent  variables  differ  considerably,  each  dependent  variable  is  standardized 
prior  to  analysis.  The  distributional  differences  of  the  92  dependent  variables  are 
examined  in  two  ways.  First,  to  see  if  the  expected  value  of  the  dependent  variables 
differ  with  the  level  of  the  independent  binary  demographic  variables,  two  sample  T-tests 
are  run  on  the  92  dependent  variables  as  a  function  of  the  independent  variables.  The 
outcome  of  this  is  a  7  x  92  matrix  of  p-values  indicating  differences  in  the  mean  value  of 
each  of  the  92  variables  as  a  function  of  the  seven  demographic  variables.  Box-plots  of 
the  92  p-values  for  each  of  the  demographic  variables  are  shown  in  Figure  4. 18. 
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Figure  4. 18  Boxplots  of  the  92  Display  Usage  and  Solution  Generation  p-Values. 

However,  in  testing  644  (7x92)  hypothesis,  both  random  statistical  noise  and 
actual  statistical  differences  are  expected  to  be  present.  If  there  is  no  difference  in  say 
LOC,  then  we  would  expect  the  92  p-values  for  LOC  to  be  uniformly  distributed  between 
zero  and  one.  By  examining  Figure  4.18  it  appears  that  the  p-values  for  all  demographic 
variables,  with  the  exception  of  POS  and  SCEN,  are  less  than  expected.  To  confirm  this, 
for  each  of  the  independent  variables,  a  Kolmorgorov-Smirnov  Goodness-of-Fit  test  for 
uniformity  is  applied  to  the  92  p-values  to  test  to  see  if  any  significance  that  was  found 
was  due  to  actual  differences  in  the  independent  variables,  and  not  due  to  random 
statistical  noise.  Secondly,  to  verify  these  results,  a  %2  test  was  run  on  the  actual 
distributional  relationships  among  the  independent  variables.  The  results  are  shown  in 
Table  4.6.  As  can  be  seen  the  variables  of  SF,  LOC,  TYPE,  and  SCEN  were  significant 
at  the  a=.05  level  and  the  variables  of  EDU  and  ROT  was  significant  at  the  a=.l  level. 
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Surprisingly,  the  variable  of  POS  was  not  a  distinguishing  trait  in  either  of  the  two  tests. 
This  gives  added  weight  to  the  earlier  conclusions  of  the  linear  regressions. 


VARIABLE 

p-Value 

KS 

x2 

SF 

0.01 

0.04 

EDU 

0.06 

0.09 

LOC 

0.01 

0.01 

POS 

0.74 

0.53 

ROT 

.05 

0.08 

SCEN 

0.01 

0.01 

TYPE 

<0.01 

0.01 

Table  4.6  KS  and  x2  p-Values  for  Display  Usage  and  Solution  Generation  Differences. 

D.  DISPLAY  DESIGN 

1.  Data  Input 

The  basic  foundation  of  human  interface  display  design  is  to  model  the  display 
after  the  expert,  or  "expected",  operators  (Bailey,  1980;  Wickens,  1992).  In  this 
experiment,  however,  the  designation  of  “expert”  operator  has  shown  a  lack  of  significant 
performance  distinction.  Therefore,  the  “successful”  operator  is  used  instead.  This  way, 
patterns  of  behavior  that  cut  across  the  demographic  differences  of  AO’s  are  exploited  for 
use  in  developing  the  underlying  principles  for  CWS  design  parameters. 

A  single  step  link  analysis  is  performed  upon  the  patterns  of  display  usage  and 
solution  generation  of  the  successful  operators.  The  patterns  of  display  usage  are 
categorized  as  belonging  to  a  10  x  10  matrix.  The  number  of  transitions  from  each  one  of 
the  10  displays  to  another  is  counted  and  then  standardized  into  subject  specific 
frequencies  with  the  constraint  that  all  of  the  100  frequencies  add  to  one.  The  10  x  10 
matrix  is  then  collapsed  to  a  4  x  4  matrix  according  to  the  coding  scheme  as  follows: 
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SONAR  (SAPBB,  TAPBB,  TAPNB),  DERIVED  (TB,  TF,  GEO,  LOS),  ENVIRON 
(OSC,  SVP),  and  SOLN  (TMA).  The  result  of  the  link  analysis  is  shown  in  Figure  4. 19. 


Figure  4.19  Successful  AO  Display  Usage  Link  Analysis. 

As  can  be  seen  the  successful  operator  utilizes  a  unique  distribution  of  display  usage  in 
the  course  of  arriving  at  a  solution.  The  three  links  of  SONAR-SONAR,  SOLN- 
DERTVED,  and  DERIVED-SOLN  account  for  more  than  50%  of  the  total  links. 

A  single  step  link  analysis  is  also  performed  upon  the  patterns  of  solution 
generation  of  the  successful  operators.  The  patterns  of  solution  generation  are 
categorized  as  belonging  to  a  4  x  4  matrix  with  the  elements  of  target  Bearing,  Range, 
Course,  and  Speed,  as  the  anchor  points.  The  number  of  transitions  from  each  one  of  the 
4  elements  to  another  were  counted  and  then  standardized  into  subject  specific 
frequencies  with  the  constraint  that  all  of  the  16  frequencies  add  to  one.  The  result  of  the 
link  analysis  is  shown  in  Figure  4.20.  As  can  be  seen,  there  are  no  obvious  biases  or 
patterns  in  the  progression  of  links.  Hence,  the  absence  of  a  distribution  is,  in  and  of 
itself,  important. 
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Figure  4.20  Successful  AO  Solution  Generation  Link  Analysis. 

Close  examination  of  the  procession  of  both  link  analysis  frequency  distributions 
show  that  in  botli  cases,  successful  subjects  follow  the  standard  contact-reporting  liturgy. 
That  is,  from  Bearing  to  Range  to  Course  to  Speed.  Deviation  from  this  progression 
appears  to  be  detrimental  to  the  outcome  of  the  engagement.  This  deviation  could  be 
responsible  for  actually  inhibiting  the  correct  decision-making  and  schema  forming 
process. 

2.  Survey  Input 

Subject  ratings  of  current  CCS  display  effectiveness  and  usefulness  are  measured 
with  both  a  five  and  a  ten-point  ordinal  ranking  scheme.  The  mean  rating  of  each  display 
is  compared  against  the  others  in  the  determination  of  the  relative  position  of  each 
display.  Again,  the  displays  are  recoded  using  the  same  methodology  as  in  the  link 
analysis.  The  results  are  shown  in  Figure  4.21  and  4.22.  In  the  measurement  of  utility, 
the  ENVIRON  displays  were  at  the  top  while  the  SOLN  displays  were  at  the  bottom.  In 
the  measurement  of  effectiveness,  the  SONAR  displays  were  at  the  top  while  the 
DERIVED  displays  were  at  the  bottom.  These  results  indicate  that  certain  CCS  displays 
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are  actually  inhibiting  the  users  SA  by  failing  to  provide  the  required  information  in  the 
proper  format.  This  also  could  inhibit  the  proper  development  of  schema's. 
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SONAR  DERIVED  ENVIRON  SOLUTION 
Figure  4.21  Subject  Response  for  the  Mean  Effectiveness  of  CCS  Displays. 


Figure  4.22  Subject  Response  for  the  Mean  Usefulness  of  CCS  Displays. 

Subject  response  to  the  open-ended  question  regarding  desires  for  the  next 
generation  of  CWS  was  mixed.  Approximately  30%  of  the  subjects  did  not  complete  the 
question  in  any  form.  Of  those  that  did,  only  10%  provided  a  written  response.  Most 
subjects,  60%,  preferred  to  interact  with  the  interviewer  in  a  “brainstorming”  session. 
The  sessions  typically  lasted  20  minutes  with  one  lasting  as  long  as  2  hours.  Their 
responses  were  manually  recorded  and  then  analyzed  for  salient  features.  A  salient 
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feature  is  defined  as  a  point,  or  issue,  of  focused  interest  by  the  subject  for  an  attribute,  or 
ability,  that  is  desired  of  the  CWS.  Additionally,  an  in-situ  drawing  was  kept  by  the 
interviewer  and  presented  to  the  subject  at  the  end  of  their  input  in  an  attempt  to  elicit 
further  input  and/or  criticism.  This  drawing  was  updated  after  each  subject’s  input  and 
evolved  in  response  to  the  subject  pool  desires  for  the  CWS.  A  frequency  analysis  of  the 
subject  generated  salient  feature  is  shown  in  Table  4.7.  As  can  be  seen  the  top  five 
attributes  were  the  availability  of  history,  multiple  tasking  capabilities,  information 
standardization,  alternative  evaluation,  and  single  point  origination  of  information. 


FEATURE 

FREQUENCY 

Availability  of  history 

83 

Multiple  tasking  capabilities 

75 

Information  standardization 

64 

Alternative  evaluation 

63 

Single  point  origination  of  information 

59 

User  defined  modes 

54 

User  defined  layout 

50 

Increased  use  of  computer  algorithms 

50 

Increased  use  of  cots 

45 

Electronic  publications 

40 

Table  4.7  Frequency  of  Salient  Features  Generated  During  Subject  Feedback. 

The  top  five  features  desired  by  the  subjects  indicate  a  desire  for  the  CWS  to  be 
fully  integrated  with  the  current  support  systems  internal  to  the  submarine  as  well  as 
providing  for  relief  from  the  current  memory  intensive  processes  now  required  of  AO’s. 
In  addition,  most  subjects  made  the  comment  that  it  would  be  advantageous  and  worth 
the  expenditure  required  to  have  displays  available  for  use  that  were  based  on  technology 
less  than  10  years  old. 
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V.  DISCUSSION  AND  CONCLUSIONS 


This  thesis  had  three  primary  objectives.  The  first  objective  was  to  explore 
possible  relationships  between  both  subject  demographic  differences  and  CCS 
employment  strategies  in  the  prediction  of  a  successful  engagement.  The  second 
objective  was  to  provide  data  in  the  development  of  a  notional  "AO  friendly"  CWS 
interface.  It  is  postulated  that  this  interface  will  allow  the  AO  to  more  accurately 
evaluate  relevant  tactical  information  with  less  effort  through  the  use  of  more  appropriate 
information  presentation  and  display  formatting.  Finally,  the  third  objective  was  to 
provide  for  principles  which  could  be  applied  in  the  areas  of  personnel  selection, 
qualification,  and  training. 

The  areas  of  HIP,  NDM,  situational  awareness,  schema  theory,  expert  vs.  novice 
performance  and  the  principles  of  display  design  were  examined  in  detail.  In  particular, 
the  military  applications,  past  and  present,  of  the  six  theories  were  discussed.  As  noted, 
most  of  the  previous  work  has  been  used  in  pilot/aircraft  interface  analysis,  and  was  not 
specific  to  submarine  AO’s.  However,  as  in  Hooper  (1994),  we  postulate  that  the  gap 
between  the  two  is  so  slight  as  to  infer  parallel  results. 

A  computer-based  simulation  was  created  and  presented  to  approximately  10%  of 
the  active  duty  submarine  AO’s.  The  subjects  were  run  though  two  of  four  pre-scripted 
initial  contact  scenarios  in  a  modified  Latin  square  design.  Two  objective  and  one 
subjective  evaluations  of  performance  were  obtained.  Solution  accuracy  at  TOF  and 
eventual,  weapon  to  target  CPA  were  continuous  objective  measures,  while  the 
determination  of  a  “successful”  engagement  was  subjective.  In  addition,  the  subjects 


63 


participated  in  a  design  survey  to  facilitate  feedback  on  not  only  the  simulation  presented 
to  them,  but  also  specific  desires  for  the  next  generation  of  CWS. 

An  exploratory  statistical  analysis  was  performed  on  the  subject-generated 
database  with  the  goal  of  determining  if  subject  demographic  differences  or  CCS  usage 
strategies  are  predictive  of  subject  performance.  Both  logistic  and  linear  regression 
techniques  were  applied  to  the  measures  of  SA  obtained  for  the  subjects.  In  the 
prediction  of  a  successful  engagement,  the  factors  of  geographical  location,  ship  type, 
and  duty  type  were  the  three  most  important  determinants.  While  these  three  attributes 
are  not  surprising  when  taken  as  a  group,  their  order  of  relative  importance  and  the 
absence  of  the  distinction  of  Expert  or  Novice  was  not  expected.  This  coupled,  with  an 
almost  zero  correlation  between  CE  and  number  of  years  at  sea,  a  synonym  for  expert 
designation,  raises  the  question  as  to  whether  a  good  AO  is  “bom  or  made”.  Analysis  of 
linear  relationships  between  time  of  engagement  vs.  CE  (Speed  vs.  Accuracy)  reveals  two 
important  observations.  First,  the  shorter  the  engagement,  the  more  accurate  the  solution 
that  is  shot.  And  second,  once  again,  expert  vs.  novice  distinction  is  negligible,  while 
geographical  location,  ship  type,  education,  and  duty  type  are  clearly  observable  effects 
in  terms  of  the  MOE's  utilized.  It  was  then  shown  via  application  of  Welsh’s  modified 
two-sample  T-test  coupled  with  Goodness-of-fit  tests  that  factors  of  SF,  EDU,  LOC, 
ROT,  and  TYPE  do  in  fact  demonstrate  variance  in  the  cognitive  progression  of  links  in 
both  display  usage  and  solution  generation.  This  lack  of  a  distinction  between  expert  and 
novice  operator  conflicts  with  the  results  expected  from  the  literature.  Perhaps,  the  core 
distinction  between  experts  and  novices  cannot  be  gleaned  from  something  as  simple  as 
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rank.  The  effects  of  the  demographic  variables  conclude  that  the  distribution  of  AO  skill 
is  not  evenly  spread  out  across  such  factors  as  geographic  location  and  ship  type. 

A  single-step  link  analysis  was  examined  on  both  display  usage  and  solution 
generation.  The  question  of  selecting  which  subject  distinctions  to  condition  on  was 
solved  by  only  examining  the  differences  between  successful  and  unsuccessful  subjects. 
The  successful  subjects  were  shown  to  follow  standard  submarine  liturgy  and  submerged 
approach  methodology  (as  explained  in  Chapter  4),  while  unsuccessful  subjects  were 
observed  to  be  going  about  their  approach  in  a  seemingly  ad  hoc  manner.  This  ad  hoc 
manner,  commonly  referred  to  as  “Easter-egging,"  exhibited  no  definable  sequence  or 
pattern  of  informational  usage.  This  further  reinforces  the  benefits  of  reliance  on 
established  methodologies  and  techniques.  Rather  than  attempt  to  develop  their  own 
individual  style,  AO's  should  instead  concentrate  on  techniques  that  have  been  proven  to 
work.  As  the  saying  goes,  "all  you  need  is  rhythm."  Finally,  results  from  the  two  link 
analyses  were  fused  with  both  subject  evaluations  of  the  current  CCS’s,  and  subject 
desire  for  attributes  of  the  next  generation  of  CCS  in  the  construction  of  a  preliminary 
CWS  design.  While  the  original  mandate  envisioned  that  subject  input  would  only  be 
made  for  the  fire  control  module  portion  of  the  CWS,  subject  dissatisfaction  with  the 
current  CCS’s  and  the  strong  desire  for  a  revolutionary  leap  in  CCS  employment, 
significantly  expanded  the  scope  of  subject  input.  The  final  product  therefore,  is  touted 
to  be  an  attempt  to  describe  the  initial  design  parameters  for  a  complete  CWS 
encompassing  all  aspects  of  AO  employment. 
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A.  NOTIONAL  CWS  DESIGN 

To  reiterate,  a  3-pronged  display  design  approach  was  utilized  in  the  development 
of  the  principles  for  the  CWS.  This  approach  combined  the  subject  survey  evaluations  of 
the  effectiveness  and  usefulness  of  the  current  CCS  displays,  the  link  analysis  extraction 
of  the  display  usage  and  solution  generation  patterns  of  the  "best"  group  of  AO's  as 
distinguished  by  the  distributional  and  categorical  relationships  identified,  and  the  top  ten 
salient  issues  generated  by  subject.  The  principles  and  parameters  developed  are 
presented  as  a  basis  upon  which  future  research  can  capitalize. 

The  physical  layout  of  the  CWS  is  founded  on  two,  side  by  side,  large  (i.e.  19-23 
inches)  computer  touch  screens.  Touch  screens  were  selected  for  their  ease  of  use  and 
quick  response  capabilities.  The  left  CWS  display,  shown  in  Figure  5.1,  consists  of  four 
fixed  modules  with  an  optional  fifth  for  submarines  with  strategic  commitments.  The 
GEO-PLOT  and  CRO  modules  share  the  same  function  as  the  modules  of  the  same  name 
in  TADMUS.  The  O/S  CONDITION  module  displays  all  actual  and  ordered  kinematic 
parameters  (course,  speed,  and  depth)  for  own-ship.  The  SVP  module  is  a  graphical 
representation  of  the  current  ocean  environment.  Ship’s  depth  would  be  displayed 
graphically  within  the  appropriate  current  SVP.  The  optional  module  for  the  left  CWS  is 
an  event  timer  for  submarines  with  strategic  commitments.  This  option  was  not  desired 
by  the  non-strategic  community  but  was  deemed  important  by  the  strategic  community. 
This  timer  would  operate  in  the  same  manner  as  the  response  manager  in  TADMUS.  The 
location  and  size  of  the  modules  was  determined  by  examination  of  the  data  and  survey 
results.  The  modules  are  arranged  to  facilitate  increased  situational  awareness  by 
allowing  for  a  more  natural  information  flow  between  the  system  and  the  operator.  This 
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then  allows  for  a  reduction  in  the  cognitive  workload  of  secondary  tasks  that  are  currently 
inhibiting  the  AO  from  forming  the  necessary  schema's.  The  now  liberated  cognitive 
resources  can  be  applied  back  to  the  AO's  primary  task,  i.e.,  approach  and  attack  of  the 
enemy. 


Figure  5.1  Left  Side  of  Notional  CWS  Display. 

While  the  left  side  of  the  CWS  is  rigid  in  structure,  the  right  side  is  user 
determined.  This  flexible  display  concept  is  primarily  an  outgrowth  of  desire  among 
AO's  that  the  CWS  be  adaptable  to  a  variety  of  situations  and  scenarios.  This  side  of  the 
CWS  is  split  into  three  modules:  an  upper,  a  lower,  and  a  series  of  selection  buttons.  The 
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right  CWS  is  display  is  shown  in  Figure  5.2.  The  selection  buttons  determine  which  of 
the  four  upper  sub-modules  (SONAR,  DAMAGE  CONTROL,  ENVIRONMENT,  and 
ERP)  and  four  lower  sub-modules  (FCS,  RADIO,  WEAPONS,  and  ENGINEERING)  are 
displayed.  The  selection  buttons  operate  such  that  only  one  of  the  upper  sub-modules 
and  one  of  the  lower  sub-modules  can  be  displayed  at  any  one  time.  The  sub-modules 
are  organized  such  that  in  most  cases,  no  two  upper  or  lower  sub-modules  will  have  to  be 
displayed  at  the  same  time. 


UPPER  DISPLAY 


LOWER  DISPLAY 


DISPLAY 

IfSilH 


Figure  5.2  Right  Side  of  Notional  CWS  Display. 


The  details  of  the  sub-modules,  with  the  exception  of  the  fire  control  system  sub- 
module  (FCS),  are  not  presented  in  this  study  due  to  space  and  time  limitations. 
However,  they  were  presented  to  the  sponsor  by  separate  correspondence.  The  FCS  sub- 
module,  shown  in  Figure  5.3,  has  four  distinct  areas  each  with  a  different  purpose. 


Figure  5.3  FCS  Sub-Module  of  Notional  CWS  Display. 

Area  one  is  a  standard  contact  report  listing  of  all  current  information  on  the 
target  of  interest.  The  order  is  top  to  bottom,  and  is  presented  in  numeric  and  bullet 
format  to  aid  in  rapid  and  detailed  recall.  Area  2,  shown  in  Figure  5.4,  itself  consists  of  4 
areas.  The  solution  area  gives  the  current  system  solution  to  the  target  selected.  The 
BRDD  (Bearing  Rate  Difference  Dots)  and  RRDD  (Range  Rate  Difference  Dots)  areas 
are  the  same  as  the  current  iteration  with  the  exception  that  they  are  standardized  in  their 
orientation.  Currently,  the  time  reference  alternates  between  top-down  and  bottom-up. 
In  the  CWS,  both  would  be  top-down  time  referenced  displays.  The  LOS  area  gives  the 
standard  3-element  representation  of  the  current  target  geometry. 
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Figure  5.4  FCS  Sub-Module  -  Area  H 

Area  three  is  the  same  as  two,  with  the  exception  that  area  two  is  keyed  to  the 
current  system  solution  and  area  three  is  keyed  to  the  current  trial  solution.  This  structure 
allows  for  the  viewing  of  both  data  representations  in  the  same  foveal  area  and  therefore 
a  reduction  of  the  memory  workload  of  the  AO.  This  also  speeds  up  the  AO's  decision¬ 
making  process  by  reducing  the  waiting  time  required  to  make  the  mental  comparisons 
between  the  current  trial  and  system  solutions.  Area  four  is  a  window  for  viewing  the 
FUSION  plot.  This  plot,  currently  undergoing  experimental  evaluation,  is  a  merging  of 
the  current  TB,  TF,  GEO,  and  ENVIRONMENTAL  plots.  This  plot  allows  for  a 
“fusion”  of  all  current  tactical  and  organic  information  pertinent  to  the  target  selected. 
One  additional  attribute  of  this  plot  that  was  requested  above  all  others  was  that  it  have  a 
“time  scaling”  feature.  In  other  words,  the  AO  would  have  the  ability  to  look  back  in 
history  to  allow  for  an  EBR  approach  to  the  target’s  intentions.  Such  a  display  might 
look  similar  to  Figure  5.5. 
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Figure  5.5  FCS  Module  -  FUSION. 

While  the  CWS’s  primary  purpose  is  to  aid  in  the  completion  of  the  AO’s  task, 
the  expanded  role  desired  by  the  subjects  raises  issues  concerning  its  physical  placement. 
The  original  CWS  was  envisioned  to  only  be  available  in  the  submarine  control  room, 
however,  with  the  added  capabilities,  it  should  be  placed  at  several  strategic  locations.  In 
particular,  repeaters  in  the  CO/XO  stateroom  and  the  officer  wardroom  would  add  in  the 
process  of  maintaining  situational  awareness  of  the  tactical  and  strategic  situation  by  the 
submarine  crew. 

This  notional  CWS  design  should  result  in  the  following:  First,  significantly 
reduced  processing  time  by  presentation  of  tactical  information  in  a  natural  and  consistent 
manner.  Second,  an  increased  CCS  functionality  by  allowing  for  a  broader  range  of 
information  to  be  available  to  the  AO.  Third,  an  increased  rate  of  user  interaction  by 
allowing  the  user  to  specify  the  type  and  time  of  information  presentation.  And  fourth, 
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an  increased  user  acceptance  of  the  system  due  to  the  fact  that  it  was  the  users  that 
determined  CWS's  form  and  function. 

B.  LESSONS  LEARNED 

Three  lessons  learned  during  the  course  of  this  research  are  worthy  of  mention: 

1.  The  use  of  a  “poor  man’s  eye-tracker”,  while  solving  many  of  the  difficulties  in 
the  data  collection  phase,  was  a  point  of  serious  consternation.  Subject 
dissatisfaction  was  very  pronounced  and  several  subjects  were  outwardly  hostile. 
The  use  of  a  head-mounted  eye-tracker,  while  expensive,  would  be  a  better 
solution. 

2.  The  simulation  employed  was  limited  in  scope  and  fidelity  by  certain  software 
design  issues  and  constraints.  Full  employment  of  available  hardware  capabilities 
to  ensure  maximum  realism  and  usefulness  must  be  a  priority  before  further  fleet 
evaluation. 

3.  Analysis  of  the  data  generated  indicates  several  areas  meriting  serious  inquiry. 
Chief  among  them  is  the  reevaluation  of  Expert/Novice  designation.  Issues 
concerning  relevant  display  design  issues  need  to  be  examined  early  in  the 
procurement  process  in  order  to  achieve  maximum  benefit. 

C.  RECOMMENDATIONS  FOR  FURTHER  RESEARCH 

Project  NEMO  shows  great  promise  for  Submarine  commanders  at  sea  operating  in 
the  high-stress,  contact  dense  and  politically  ambiguous  environments  that  the  future 
surely  promises.  However,  five  further  research  issues  merit  consideration: 
l.  Use  of  the  database  for  further  analysis.  Only  a  few  data  analysis  avenues  and 
hypotheses  were  examined  in  this  thesis.  Future  work  by  NUWC,  working  in 
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conjunction  with  staff  and  students  at  NPS,  should  continue  to  explore  the  wealth 
of  data  collected  during  the  course  of  this  experiment.  In  addition,  several  areas 
were  not  explored  due  to  the  limitations  of  subject  pool  size,  composition,  and 
statistical  significance.  Continued  data  collection,  perhaps  from  East  Coast  based 
submarine  AO’s,  would  provide  the  necessary  database  to  alleviate  this. 

2.  Continued  CWS  development.  NUWC  and  ONR  should  seriously  consider  the 
principles  developed  for  the  preliminary  CWS  design  proposed  for  further 
evaluation  and  possible  implementation  in  tools  such  as  Micro-SAINT  (Wickens, 
1992),  with  the  eventual  goal  of  formal  presentation  to  the  NSSN  program  office. 
This  preliminary  design,  constructed  for  the  first  time  by  actual  operators,  should 
result  in  improved  AO  situational  awareness,  increased  tactical  proficiency,  and 
greater  overall  operator  satisfaction. 

3.  Cognitive  modeling  of  novice  system  users.  As  stated  earlier,  the  time  constraints 
imposed  by  rapidly  unfolding  scenarios  will  surely  be  measured  in  minutes, 
possibly  seconds,  rather  than  hours  or  days.  The  systems  must  be  able  to  be  used 
effectively  by  the  all  levels  of  operators,  since  it  is  the  Junior  Officer  who  most 
likely  will  have  the  watch  when  hostilities  commence.  This  would  entail  a 
reevaluation  of  the  system  from  the  "novice"  point  of  view  in  order  to  facilitate 
maximum  throughput  of  key  tactical  information.  A  mode  command  or  toggle 
device  installed  into  the  system  could  be  used  to  shift  between  levels  of  user  skill. 

4.  Use  of  NEMO  for  Command  Level  training.  The  nature  of  command  in  a 
submarine  control  room  is  that  the  users  operate  in  a  windowless  environment 
devoid  of  external  stimuli  or  reference  to  the  outside  world.  This  environment 
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can  be  replicated  quite  easily  via  simulators.  The  simulators,  based  at  major  shore 
installations  as  separate  entities  or  onboard  the  units  themselves  in  the  form  of 
“canned  training  scenarios”,  would  provide  for  the  proper  training  of  current  and 
future  commanders.  The  simulators  could  also  provide  some  sort  of  diagnostic 
decision  making  performance  measures  that  could  be  compiled  automatically  to 
indicate  the  skill  of  the  user. 

5.  Use  of  NEMO  for  Command  selection.  Selection  for  command  at  sea,  while 
monumental  for  the  selectee,  is  at  best  a  vague  and  uncertain  process  for  selection 
boards.  Frequently  board  members  have  little  or  no  personnel  experience  or 
knowledge  of  the  selectee  and  so  they  base  their  decision  on  paper  reports, 
second-hand  recommendations,  and  “sea  stories."  The  CWS  developed  by 
Project  NEMO  could  be  used  to  supplement  the  board's  decision  by  testing  the 
potential  commander's  innate  decision  making  process.  That  process  could  then 
be  categorized  to  see  if  the  selectee  possesses  the  correct  cognitive  processes 
required  of  “successful”  commanders.  The  selectee  would  be  run  through  a 
scenario  specifically  crafted  by  the  board  and  his,  or  her,  outcome  would  aid  in 
the  determination  of  adequacy  for  command.  Alternatively,  since  cognitive 
abilities  are  for  the  most  part  "bom"  and  not  made,  the  systems  could  be  used  for 
program  entrance  selection  across  the  entire  range  of  warfighter  roles. 

In  conclusion,  this  thesis  has  provided  insight  into  the  cognitive  processes  of  the 
AO.  The  proposed  CWS,  in  addition  to  assisting  the  AO  during  tactical  engagements, 
has  applications  in  the  selection,  initial  and  continuing  training,  and  performance 
evaluation  of  the  AO.  It  is  hoped  that  issues  raised  and  conclusions  drawn  in  the  course 
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of  this  experiment  will  stimulate  further  research  in  the  application  of  human  factors 
principles  to  modem  submarine  design. 
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APPENDIX  A:  TADMUS 


TADMUS1,  which  is  currently  in  its  second  iteration  of  development,  is  entering 
the  human  factors  test  and  evaluation  phase.  The  program  managers,  after  a  full  testing 
of  the  first  iteration,  significantly  changed  the  layout,  design,  and  cognitive  models  that 
drive  TADMUS.  Figure  A1 . 1  shows  the  current  screen  layout  of  TADMUS.  TADMUS 
consists  six  distinct  modules  each  with  a  specific  purpose:  Geo-Plot,  Multi-CRO  Access 
Panel,  Track  Profile,  Response  Manager,  Track  Summary,  and  Basis  for  Assessment. 


Fig  Al.l  TADMUS  Display. 

Geo-Plot 

The  Geo-Plot  module  is  the  primary  display  for  the  command  team  (see  Figure 
A1.2).  It  uses  standard  Naval  Tactical  Data  Exchange  (NTDS)  color  symbology  as  well 
as  system  track  numbers  to  code  all  air,  surface,  and  subsurface  elements  in  the 
battlespace.  The  map  can  be  expanded  or  enhanced  to  show  any  level  of  clarity  available 


1  The  following  discussion  has  been  excerpted  from  Dr.  Hutchins  writings  on  TADMUS  in  1995/1996. 
Hutchins,  S.G.  (1996).  Principles  for  Intelligent  Decision  Aiding.  (SPAWAR  TM 1718).  DON:  Coronado, 
CA.  ~ 

Hutchins,S.G.  &Rummel,  D.K  (1995).  A  Decision  Support  System  for  Tactical  Decision-making  Tinder 
Stress.  Proceedings  of the  first  International  symposium  on  Command  and  Control  Research  and 
Technology.  Montery,  CA. 
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in  Joint  Operational  Tactical  Exchange  (JOTS).  This  ranges  from  general  area  maps  to 
specific  city  street  maps.  In  addition,  users  can  overlay  maps  to  increase  detail  or  “de- 
saturate”  them  to  reduce  clutter.  The  Geo-Plot  can  also  show  weapons  information  such 
as  range  brackets,  areas  of  uncertainty  (AOU),  and  projected  tracks  (incoming  and 
outgoing)  to  allow  own  ship  to  rapidly  assess  the  criticality  of  a  threat. 


Fig  A1.2  Geo-Plot  Display. 


Multi-CRO  Access  Panel 

The  Multiple-Character  Read  Out  (CRO)  Access  Panel  module  displays  all  of  the 
relevant  information  concerning  target  identification  and  kinematics  (see  Figure  A1.3). 
In  addition,  it  allows  for  the  most  critical  tracks  to  be  monitored  without  additional 
interaction  from  the  user  with  the  system.  The  buttons  are  all  identical  with  priority 
assigned  from  left  to  right  in  order  of  decreasing  importance.  The  CRO’s  are  color  coded 
to  show  threat  status  and  “blink”  when  pertinent  track  information  is  available. 
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Fig  A1.3  Multi-CRO  Display. 

Track  Profile 

The  Track  Profile  module  complements  the  Geo-Plot  by  displaying  a  side  view  of 
track  altitude  and  range  from  own  ship  (see  Figure  A1.4).  Information  is  available  as  to 
where  the  track  is,  what  the  track  has  been  doing  since  tagging,  when  the  track  is  within 
its  presumed  weapons  range,  and  when  own  ship  is  within  weapons  range.  The  module 
also  allows  for  own  ship  to  evaluate  multiple  weapons  type  responses  as  well  as 
projecting  weapons  effectiveness  sphere's  from  other  friendly  assets  in  the  battlespace. 
This  feature  will  be  critical  for  the  notion  of  future  “Joint  Battle-Groups”. 


Fig  A1.4  Track  Profile  Display. 

Response  Manager 

The  Response  Manager  module  provides  a  Gantt  chart  type  display  showing  a  set 
of  pre-planned  actions  as  well  as  the  optimal  range  for  execution  for  the  target  of  interest 
(see  Figure  A1.5).  These  pre-planned  actions  are  intended  to  be  based  on  the  current 
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Order  of  Battle  that  the  commander  is  operating  under  and  can  be  updated  as  frequently 
as  required 
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Fig  A1.5  Response  Manager  Display. 


Track  Summary 


The  Track  Summary  module  supplements  the  Multi-CRO  panel  by  providing  a 
detailed  summary  of  all  current  and  historical  classification  and  kinematic  data  for  the 
selected  track  (see  Figure  A1.6).  It  also  provides  for  the  display  of  ancillary  data,  such  as 
intelligence,  Electronic  Warfare  (EW),  and  Identification  Friend  or  Foe  (IFF). 


Fig  A1.6  Track  Summary  Display. 


Basis  for  Assessment 


The  Basis  for  Assessment  module  provides  the  command  team  with  detailed  lists 
of  evidence  for  and  against  the  current  assessment  of  the  selected  track  (see  Figure  A1.7). 
It  also  presents  presumed,  but  yet  unknown,  information  and  implicit  assumptions  being 
made  in  accepting  the  classification  of  the  contact  as  a  potential  non-threat,  threat,  or 
unknown. 


Fig  A1.7  Basis  for  Assessment  Display. 
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APPENDIX  B:  EXPERIMENTAL  PROTOCOL  AND  INFORMED  CONSENT 


EXPERIMENTAL  PROTOCOL 


Ned  Experimental  Protocol  (total  duration  ~2  -  3  hours) 

Conventions 

“=>“  is  used  before  actions 
|  Things  in  this  format  are  to  be  read  aloud  verbatim. 

|  Things  in  this  format  are  to  be  read  aloud  verbatim  and  stressed 


Things  in  this  format  are  notes  to  the  experimenter 
Experiment  Setup 

=>  Put  a  fresh  tape  in  the  VCR  and  audio  recorder,  batteries  in  the  audio  recorder, 
and  make  sure  the  connections  are  all  working 

=>  Be  sure  to  disable  any  applications  that  run  on  a  scheduled  basis,  such  as  Norton 
FileSaver,  mail  programs,  etc.,  these  will  disrupt  the  timing  in  Ned. 

=>  Have  the  tic-tac-toe  HyperCard  stack  up,  running,  and  hidden. 

=>  Have  the  TTT  video  movie  up,  running,  and  hidden. 

=>  Have  Ned  (vl.5.5)  up,  running,  and  hidden. 

=>  Be  sure  to  have  the  informed  consent  forms  handy 

Experiment  Briefing  (5-10  min.) 

The  basic  idea  here  is  to  give  the  subject  a  brief  overview  of  the  purpose  of  the  study  and 
what  will  be  expected  of  them. 

=>  Read: 

This  is  an  unclassified  system  being  used  for  basic  research.  This  exercise  will  be 
audio/video  recorded.  Please  keep  all  remarks  unclassified. 

Today  I  will  ask  you  to  participate  in  two  scenarios  using  a  simplified  command 
workstation  simulation.  The  purpose  of  this  experiment  is  to  gather  data  on  how 
command  might  use  information  that  is  typically  available  in  the  Combat  Control 
System.  As  you  know,  the  new  SSN  that  is  being  planned  will  have  a  dedicated  command 
workstation  (CWS)  to  support  the  Approach  Officer.  As  this  is  a  new  console,  there  is 
little  known  about  how  such  a  workstation  will  be  used.  This  experiment  is  one  in  a  series 
of  studies  to  predict  the  answer  to  that  question.  One  thing  we  do  know,  gleaned  from  the 
experience  of  many  submarine  officers,  is  that  the  CWS  will  require  different  displays 
than  those  currently  available  on  operator  consoles.  Your  participation  will  help 
determine  the  content  of  those  displays. 

In  this  exercise,  your  role  will  be  that  of  the  AO. 
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Data  collection  in  this  experiment  depends  on  making  your  information  needs  and  usage 
visible.  A  large  part  of  our  task  is  to  understand  what  information  you  need  and  when  you 
need  it.  Hence,  I  will  ask  you  to  talk  aloud  as  you  work.  I  will  explain  this  in  more  detail 
and  give  you  an  opportunity  to  practice  doing  this  shortly.  Videotapes  will  be  made  of 
every  scenario  and  computerized  data  is  being  collected  as  well. 

In  accordance  with  requirements  of  NUWC's  Committee  for  the  Protection  of  Human 
Subjects,  strict  procedures  have  been  established  to  preserve  the  confidentiality  of 
participants.  Outside  of  the  experimenters,  no  one  can  gain  access  to  information  that 
would  reveal  your  identity.  If  you  have  any  concerns  that  I  should  address,  please  ask 
them  now. 

=>  Hand  them  the  Informed  Consent  forms  and  ask  them  to  read  and  sign 

Talk-aloud  training  (5  - 10  min.) 

The  idea  here  is  to  give  the  subject’s  some  specific  instructions  on  what  sorts  of  things 
we  are  going  to  want  them  to  report  and  also  to  get  them  comfortable  with  talking  aloud 
while  working.  To  this  end,  the  subject  is  given  instructions  ala  Ericsson  &  Simon  and  a 
practice  task  to  work  on  with  tic-tac-toe.  The  subject  plays  a  couple  of  games  and  is 
given  feedback  on  the  quality  of  the  protocols 
=>  Bring  up  the  tic-tac-toe  game  and  TTT  video 

Read: 

As  you  recall,  I  said  that  you  are  to  talk  aloud  during  this  study.  This  may  require  some 
example  and  practice,  so  lets  first  watch  the  computer  to  it  and  then  you’ll  play  the 
computer. 

=>  Play  TTT  for  one  approximately  one  minute. 

=>  Read: 

To  reiterate,  what  I  mean  by  talk  aloud  is  that  I  want  you  to  say  out  loud  everything  that 
you  say  to  yourself  silently.  Just  act  as  if  you  are  alone  in  the  room  speaking  to  yourself. 
If  you  are  silent  for  any  length  of  time,  I  will  remind  you  to  keep  talking  aloud.  Do  you 
understand  what  I  want  you  to  do? 

You  may  begin  playing  when  you  are  ready 

=>  If  silent  for  more  than  10-15  seconds  say  “keep  talking” 

=>  Give  feedback  after  each  game 

=>  Continue  until  subject  does  not  need  to  be  reminded  more  than  once  during  the 
entire  game. 

=>  Quit  HyperCard  and  MoviePlayer. 

=>  You  should  be  at  the  15-minute  point 

Introduction  to  Ned  (25  -  30  min.) 

Freeplav  Mode 

The  goal  of  this  part  is  to  provide  the  subjects  with  enough  knowledge  about  the  interface 
to  get  good,  clean  data  from  the  experiment  sessions.  Also,  because  data  collection  will 
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occur  with  information  covered  by  the  gray  boxes,  we  can  avoid  extraneous  mouse  clicks 
by  ensuring  that  subjects  know  where  to  find  the  information  they  are  seeking. 

Prompted  Practice 

=>  Bring  up  Ned  and  read: 

I  will  now  show  you  the  simulated  combat  control  system  (CCS).  With  it  you  can 
monitor  all  normal  command  functions,  ship's  control,  Sonar,  and  TMA  (MATE,  Geo  Sit, 
and  plots).  You  can  also  order  changes  in  own  ship's  speed,  course,  and  depth.  As  this  is  a 
simulation  designed  for  experimental  purposes  and  unclassified,  there  are  some 
limitations  and  unrealistic  elements,  which  I  will  note  as  we  go  along. 

I  will  introduce  this  simulation  in  two  phases.  First  I  will  walk  you  through  all  of  the 
displays  and  give  you  an  opportunity  to  ask  questions  about  the  contents  of  the  displays 
and  to  practice  using  the  interface  controls.  I  will  then  switch  the  simulation  over  to  data 
collection  mode,  which  is  the  mode  in  which  you  will  be  working  for  the  main  part  of  the 
study,  to  let  you  practice  and  explore  the  simulation  under  those  conditions. 

Please  feel  free  to  ask  questions  as  we  work  through  the  displays,  I  want  to  make  sure 
you  understand  how  all  of  thee  displays  work  before  we  start  collecting  actual  data. 

I  also  want  to  remind  you  that  this  is  an  unclassified  system  being  used  for  basic  research. 

This  exercise  will  be  audio/video  recorded  so  please  keep  all  remarks  unclassified. 

=>  Make  sure  the  Practice  scenario  is  selected  in  the  pop-up  menu 

=>  Press  the  “Start”  button 

=>  Let  the  subject  have  control  of  the  computer 

=>  Ned  will  display  the  “Problem  Description”  and  then  “Status  Report”  windows. 
Read  the  text  on  these  displays  aloud  and  press  the  buttons  to  continue. 

=>  When  it  gets  to  OS  control,  read  the  following  and  point  to  each  thing  as  you 
describe  it. 

This  window  allows  you  to  control  own  ship.  These  fields  contain  the  ordered  course, 
speed,  and  depth.  To  change  course,  select  the  field,  type  the  new  ordered  course,  select 
the  direction  of  the  turn,  then  click  on  this  initiate  change  button.  These  fields  above 
display  the  current  actual,  course  speed  and  depth. 

Go  ahead  and  turn  right  to  a  bearing  of  150,  and  slow  down  to  7  knots. 

=>  While  OS  is  maneuvering,  read: 

|  Note  that  the  fields  up  top  change  as  OS  changes,  while  the  fields  below  remain  constant. 

=>  When  maneuver  is  complete,  read: 

This  is  the  display  navigation  palette.  You  will  be  able  to  look  only  at  one  display  at  a 
time.  To  look  at  a  display,  simply  press  the  appropriate  button.  We  will  first  go  to  the 
broadband  spherical  waterfall  display.  Go  ahead  and  click  the  button  labeled  “BB- 
Sphere”. 

=>  When  BB-Sphere  comes  up,  read: 
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This  is  the  waterfall  window  for  the  broadband  sphere.  It  displays  true  bearing  from  0  to 
359  degrees  and  is  divided  into  two  parts.  In  the  upper  part  the  integration  time  is  twenty 
seconds.  In  the  lower  part  it  is  ten  minutes.  OS  course  is  represented  by  this  blue  line  in 
the  upper  part. 

As  you  can  see  we  have  a  trace  here  at  a  bearing  of  about  220,  which  corresponds  to  the 
merchant  contact  from  our  status  report.  To  put  a  tracker  on  a  target,  just  position  the 
cursor  over  the  track  until  it  changes  shape,  then  click.  Go  ahead  and  set  a  tracker  on  this 
target. 

=>  When  tracker  is  set,  read: 

As  you  can  see,  the  information  on  the  track  is  displayed  here.  It  assigns  a  sierra  number, 
in  this  case  sierra  1,  and  gives  you  the  true  bearing,  SNR,  and  DE. 

Along  with  a  broadband  sphere,  you  also  have  a  broadband  towed  sonar  sensor.  If  you 
don’t  have  any  questions  about  this  display,  go  ahead  and  click  the  button  labeled  “BB- 
Towed”. 

=>  When  BB-Towed  comes  up,  read: 

Like  the  spherical  sensor,  the  broadband  towed  display  is  also  a  waterfall  display  with  20 
seconds  of  integration  time  in  the  top  part  and  10  minutes  of  integration  time  in  the 
bottom  part.  Unlike  the  bb-sphere,  this  display  plots  conical  angle  from  0  to  180  degrees. 
You  set  a  tracker  in  the  same  manner  as  in  the  other  display  -  just  move  the  cursor  over 
the  trace  until  it  changes,  and  then  click.  Go  ahead  and  set  a  tracker  on  this  trace. 

=>  When  tracker  is  set,  read: 

As  before,  the  information  on  the  track  is  displayed  here.  It  assigns  a  sierra  number,  in 
this  case  sierra  1,  and  gives  you  the  conical  angle,  both  ambiguous  bearings  and  SNR.  In 
this  field  below,  you  can  check  the  status  of  the  towed  array.  As  you  can  see,  the  array  is 
currently  stable. 

The  third  sonar  sensor  available  to  you  is  narrowband  towed.  If  you  don’t  have  any 
questions  about  this  display,  go  ahead  and  click  the  button  labeled  “NB-Towed”. 

=>  When  NB-Towed  comes  up,  read: 

This  display  shows  frequency  on  the  horizontal  axis  and  conical  angle  on  the  vertical 
axis.  The  frequencies  displayed  here  are  completely  fabricated  due  to  the  unclassified 
nature  of  the  simulation.  The  key  point  is  that  frequencies  to  the  left  are  where  you  are 
likely  to  find  a  merchant  contact  and  frequencies  to  the  right  are  where  you  would  find  a 
submerged  contact.  As  you  can  see,  we  have  one  of  each  currently  on  sonar.  Go  ahead 
and  set  trackers  on  both  of  these  targets. 

=>  When  tracker  is  set,  read: 

As  you  can  see,  the  information  on  the  tracks  is  displayed  off  to  the  right,  and  includes 
sierra  number,  conical  angle,  both  ambiguous  bearings  and  SNR.  The  status  of  the  towed 
array  is  shown  below. 
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Data  from  these  three  sensors  are  sent  to  several  other  displays,  including  a  time  bearing 
plot,  time  frequency  plot,  and  TMA  Lets  start  with  time  bearing,  go  ahead  and  press  the 
TimeBmg  button  on  the  palette. 

=>  When  TBR  comes  up,  read: 

As  you  can  see,  this  display  plots  0  to  360  degrees  of  bearing  over  a  20  minute  period. 

To  use  this  display,  first  select  the  sensor  for  which  you  would  like  to  see  data.  Go  ahead 
and  click  the  boxes  labeled  bb-spherical  and  nb-towed. 

=>  When  boxes  are  checked,  read: 

The  traces  correspond  in  color  to  the  label  on  the  boxes,  and  the  size  of  the  boxes 
correspond  to  target.  The  larger  boxes  are  Sierra  1  and  the  smaller  ones  are  Sierra  2.  To 
get  the  bearing  rate,  move  the  magnifying  glass  over  the  box  of  interest  and  read  the 
bearing  rate  from  the  field  below.  Note  that  true  bearing  is  also  available.  Each  box 
represents  20  seconds  of  data,  so  the  bearing  and  bearing  rates  you  are  reading  have  been 
averaged  over  20  seconds. 

The  narrowband  sensors  send  data  to  the  time  frequency  plot,  which  we  will  look  at  next. 
If  you  don’t  have  any  questions,  go  ahead  and  press  the  TimeFreq  button  on  the  palette. 

=>  When  TFP  comes  up,  read: 

This  display  plots  a  40  hertz  span  of  frequency  over  a  10  minute  period.  Each  box 
represents  data  averaged  over  6  seconds.  This  display  works  in  a  manner  very  similar  to 
the  time  bearing  window.  First  you  select  a  target  from  the  pop-up  menu  over  here,  then 
you  move  the  magnifying  glass  over  the  box  to  read  the  frequency.  The  time  that  is 
displayed  is  the  scenario  time  that  the  box  was  plotted. 

If  you  have  no  questions  about  this  display,  we  will  move  on  to  the  target  motion  analysis 
window.  Go  ahead  and  click  the  TMA  button. 

=>  When  TMA  comes  up,  read: 

This  display  allows  you  to  work  one  solution  at  a  time  for  each  target  and  sensor.  You 
select  the  target/sensor  from  this  pop-up  menu.  Go  ahead  and  select  the  broadband 
spherical  solution  for  sierra  1. 

=>  When  sierral-bbs  solution  appears,  read: 

As  you  can  see,  the  system  has  provided  you  with  a  starting  solution.  This  solution  is 
displayed  in  the  solution  data  box.  It  assigns  the  solution  an  ID#,  and  displays  the  range, 
bearing,  course,  speed,  bearing  rate,  and  range  rate,  as  well  as  the  age  of  the  tracker 
sending  data  to  TMA. 

This  solution  is  plotted  on  the  geosit  below.  You  have  a  choice  of  four  zoom  levels  via 
this  pop-up  menu.  The  number  on  the  menu  corresponds  to  the  outermost  circle  on  the 
geosit.  Go  ahead  and  select  the  40  kyd  scale.  The  target  is  plotted  as  a  circle  with  a  course 
speed  vector.  A  bearing  line  is  plotted  for  the  spherical  array,  and  a  parabola  for  the 
towed  arrays. 

Over  to  the  left  are  plots  of  the  solution  bearing  and  the  error  stack.  These  can  plot  up  to 
15  minutes  of  data  at  a  time. 
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|  To  change  the  solution,  press  the  tweak  solution  button.  Go  ahead  and  press  it  now. 

=>  When  tweak  solution  dialog  appears,  read: 

In  this  dialog,  you  can  modify  the  solution  range,  bearing,  course  and  speed.  To  change 
parameters,  simply  type  the  new  values  in  the  fields  and  press  either  the  preview  or  the 
done  button.  The  preview  button  will  show  the  changes  in  the  TMA  window,  but  keep 
the  tweak  solution  dialog  open.  The  done  button  will  make  the  changes  and  close  the 
dialog.  Go  ahead  and  change  a  few  parameters  and  then  hit  the  preview  button. 

=>  When  new  solution  displayed,  read: 

As  you  can  see,  the  new  solution  appears  in  all  parts  of  the  time  window.  If  you  choose 
to  keep  the  solution,  press  the  done  button.  If  you  want  to  modify  it  further,  you  can  type 
new  parameters  in  the  fields  and  then  hit  preview  or  done.  If  you  want  to  go  back  to  the 
solution  you  had  before  you  hit  the  tweak  solution  button,  press  cancel. 

Note  that  whenever  the  tweak  solution  dialog  is  open,  you  will  be  unable  to  do  anything 
else  in  the  window,  so  be  sure  to  close  the  dialog,  either  by  hitting  cancel  or  done,  before 
attempting  to  do  anything  else.  Go  ahead  and  hit  the  done  button  now. 

If  there  are  no  questions  on  the  TMA  window,  lets  move  on  to  the  Geosit  window.  Go 
ahead  and  click  the  GEOSIT  button  now. 

=>  When  GEOSIT  comes  up,  read: 

This  display  will  allow  you  to  see  solution  plots  on  both  targets  simultaneously.  To  view 
a  solution,  simply  select  it  from  one  of  these  two  pop-up  menus.  The  top,  black  menu  is 
for  sierra  1  and  the  lower,  red  menu  is  for  sierra  2.  As  you  can  see,  there  is  also  a  menu  to 
select  the  scale  of  the  plots,  again,  with  the  number  on  the  menu  corresponding  to  the 
outermost  ring. 

Go  ahead  and  select  the  broad  band  spherical  solution  for  sierra  1  and  the  narrowband 
towed  solution  for  sierra  2. 

=>  When  solutions  displayed,  read: 

As  you  can  see,  each  target  is  plotted  as  a  circle  with  a  course  speed  vector.  The  colors 
of  the  targets  correspond  to  the  colors  on  the  pop-up  menus,  with  sierra  1  in  black  and 
sierra  2  in  red.  Note  that  this  display  also  plots  bearing  lines.  It  will  plot  a  maximum  of  5 
bearing  lines  each  2  minutes  apart. 

If  there  are  no  questions  on  this  display,  let’s  move  on  to  the  line-of-sight  display.  Go 
ahead  and  hit  the  LOS  button  now. 

=>  When  LOS  comes  up,  read: 

This  is  a  passive  display  in  that  it  has  no  data  being  sent  to  it.  You  will  have  to  provide 
the  values.  In  order  to  use  it,  enter  values  into  each  of  these  fields  and  then  press  the  plot 
button.  Go  ahead  and  enter  in  some  values  now.  Don’t  worry  it  they  are  correct  or  not 
for  now. 

=>  When  LOS  plotted,  read: 

|  As  you  can  see,  the  target’s  course  is  plotted  above  and  ownship’s  is  below. 


96 


If  there  are  no  questions  about  this  display,  let’s  move  on  to  the  final  display,  the  sound 
velocity  profile.  Go  ahead  and  hit  the  SVP  button  now. 

=>  When  LOS  comes  up,  read: 

This  display  is  inanimate  and  simply  plots  the  current  SVP.  There  are  no  actions  that  can 
be  performed  on  this  display. 

Do  you  have  any  questions  on  this  or  any  other  display? 

Okay,  then  we  will  move  on  to  the  next  phase  of  the  study  where  I  show  you  some  of  the 
features  of  the  simulation  in  its  experiment  mode.  Go  ahead  and  click  the  firing  point 
procedures  button  to  stop  the  simulation. 

=>  Take  control  of  the  computer.  When  the  main  control  window  comes  up  click 
“Stop”  then  “Reset”. 

Experiment  Mode 

The  goal  here  is  to  familiarize  the  subject  with  the  experiment-specific  stuff  -  primarily 
using  the  gray  boxes  and  talking  aloud  while  working.  It  is  important  to  describe  in  some 
detail  the  use  of  the  boxes  in  the  TBR  and  TFP,  as  they  work  a  little  differently  than  they 
do  in  other  displays. 

After  the  demonstration,  the  subject  is  instructed  to  try  to  determine  a  solution  for  the 
scenario  and  to  talk  aloud  while  doing  so.  Provide  feedback  on  the  quality  of  the 
protocols.  When  the  subject  is  providing  good  quality  protocols  and  is  comfortable  with 
the  use  of  the  gray  boxes  he  is  ready  for  the  data  collection  phase. 

=>  Select  the  “Run  experiment”  check-box  and  press  the  “Configure...”  button. 

=>  Enter  your  initials,  the  subject  number,  and  trial  number  of  “0”  in  the  Setup 
window 

=>  Press  the  “Create  Data  File...”  button. 

=>  Make  sure  the  “Collect  Biographical  Data”  check  box  is  NOT  checked  and  the 
“Wash-up”  one  is.  Press  the  “Done”  button  when  finished. 

=>  Press  the  “Start”  button  to  bring  up  the  problem  description  window 

=>  Let  the  subject  have  control  of  the  computer 

Prompted  practice 
=>  Read: 

Now  we  are  going  to  look  at  experiment  mode.  There  will  be  a  few  differences  from  what 
I  just  showed  you.  We  are  running  the  same  practice  problem  that  we  did  the  last  time  so 
the  problem  description  and  scenario  description  are  the  same.  Just  go  ahead  and  press 
the  View  Status  report  and  Begin  buttons  on  the  first  two  windows. 

=>  When  OSC  window  comes  up,  read: 

One  major  difference  is  that  all  data  fields  are  covered  by  gray  boxes  until  you  put  the 
cursor  in  them  and  click  the  mouse  button.  These  fields  are  exactly  the  same  as  the  ones 
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I  you  saw  before,  they  are  just  covered  up  by  the  boxes.  Go  ahead  and  click  on  the  box 
over  ordered  OS  course. 

=>  When  OS  course  field  is  visible,  read: 

I  When  you  move  the  mouse  out  of  the  field,  the  information  is  covered  back  up.  Go  ahead 
and  move  the  mouse  out  of  the  field. 

=>  When  OS  course  field  covered  back  up,  read: 

Some  of  the  boxes  are  very  small,  so  tiy  not  to  move  the  mouse  out  unintentionally. 

The  fields  do  receive  data,  but  the  values  are  not  visible  until  you  click  on  them. 
Likewise,  you  also  won’t  be  able  to  enter  a  value  into  a  field  unless  it  is  visible. 

Generally  speaking,  you  will  be  able  to  look  at  only  one  thing  at  a  time. 

Go  ahead  and  turn  to  the  right  to  1 50  degrees,  and  then  watch  the  values  change  in  the 
course  status  field  above. 

=>  When  OS  has  completed  the  maneuver,  read: 

Okay,  lets  go  to  the  narrowband  towed  display  so  you  can  practice  setting  a  tracker  in 
experiment  mode. 

=>  When  NB-Towed  comes  up,  read: 

You  won’t  be  able  to  set  a  tracker  unless  you  uncover  the  waterfall  display  first.  Go 
ahead  and  click  on  the  waterfall  box  and  set  a  tracker  on  the  merchant. 

=>  When  tracker  has  been  set,  read: 

As  you  can  see,  the  information  on  the  tracker  is  also  covered  up.  To  see  the  conical 
angle,  bearings,  or  SNR,  you  will  have  to  click  on  the  appropriate  gray  box.  Go  ahead 
and  click  on  these. 

The  fields  in  the  displays  all  work  the  same  as  this  except  for  the  Time  Bearing  and  Time 
Frequency  displays.  Let’s  go  to  the  time  bearing  display  first. 

=>  When  TBR  comes  up,  read: 

|  As  before  you  will  first  have  to  select  a  sensor.  Go  ahead  and  click  nb-towed. 

=>  When  narrowband-towed  data  plotted,  read: 

I  You  also  have  to  choose  whether  you  want  to  see  bearing  or  bearing  rate.  You  will  only 
be  able  to  see  one  at  a  time.  Go  ahead  and  click  on  the  box  next  to  bearing  rate. 

=>  When  bearing  rate  box  uncovered,  read: 

Unlike  the  other  gray  boxes,  this  one  will  stay  open  when  the  mouse  leaves.  This  is  so 
you  can  move  the  magnifying  glass  around  to  get  information  from  the  data  plotted  on 
screen. 

Go  ahead  and  move  the  magnifier  over  some  data. 

=>  When  subject  has  done  this,  read: 
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I  This  box  will  stay  open  until  you  select  bearing.  Go  ahead  and  click  on  the  box  next  to 
true  bearing. 

=>  When  bearing  box  uncovered,  read: 

As  you  can  see,  bearing  rate  is  now  covered,  so  you  can  only  see  the  true  bearing. 

The  TFP  display  works  very  much  like  this.  Let’s  take  a  look  at  that  one  now. 

=>  When  TFP  comes  up,  read: 

The  frequency  and  time  fields  act  in  the  same  manner  as  bearing  rate  and  true  bearing 
fields  in  the  TBR  window.  Go  ahead  and  select  S001  from  the  pop-up  menu  and  then 
click  the  box  next  to  freq. 

=>  When  subject  has  done  this,  read: 

I  Now  when  you  move  the  magnifying  glass  over  the  data,  you  can  read  the  frequency  but 
not  the  time. 

Free  Practice 
=>  Read: 

Now  it  is  your  turn  to  practice  using  the  simulation  in  this  mode.  Feel  free  to  ask 
questions  as  you  work.  I  would  also  like  you  to  practice  talking  aloud  as  you  do  this. 
When  you  are  comfortable  with  the  system  and  feel  you  have  a  reasonable  solution  on  the 
targets,  select  this  button  labeled  FPP.  It  stops  the  scenario  and  takes  you  to  a  display  for 
a  hot  wash-up. 

=>  Watch  the  subject  interact  with  the  system,  answering  any  questions  he  has 

Wash-up  Prompted  Practice 

=>  When  the  GEO  DRAW  window  appears,  read: 

Here  we  will  want  you  to  indicate  what  happened  during  the  scenario.  If  there  were 
discrepancies  between  your  solution  and  the  system,  you  should  indicate  your  solution 
rather  than  the  system  solution. 

Here  is  how  it  works.  First  you  select  a  contact,  in  this  case,  Sierra  1 A  is  already  selected 
for  you.  So  go  ahead  and  click  in  on  the  geosit  the  bearing  and  range  where  you  think  the 
merchant  was  at  the  end  of  the  leg. 

=>  When  the  subject  has  dropped  the  target,  read: 

The  next  mouse  click  draws  a  course-speed  vector  from  the  center  of  the  target  to 
wherever  you  click.  Go  ahead  and  do  this  now. 

=>  When  the  subject  has  drawn  the  vector,  read: 

I  A  third  mouse  click  draws  a  line  indicating  of  range  uncertainty  surrounding  the  target. 
Click  anywhere  along  the  vector  from  the  center  of  the  geosit  to  the  target. 

=>  When  the  subject  has  drawn  range  uncertainty,  read: 

|  As  you  can  see,  this  draws  a  gray  line  surrounding  the  target. 
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You  may  indicate  bearing  ambiguity  by  selecting  a  second  version  of  the  same  contact 
number,  using,  for  example,  1 A  for  the  more  likely  ambiguous  bearing  and  IB  for  the 
other  ambiguous  bearing.  Select  sierra  lb  from  the  pop  up  menu  and  draw  the  ambiguous 
bearing  on  the  merchant,  including  a  course  speed  vector  and  range  uncertainty.  Don’t 
worry  if  it  is  correct  or  not,  this  is  only  practice. 

=>  When  the  target  has  drawn  sierra  IB  on  the  geosit,  read: 

If  you  want  to  delete  a  target,  select  the  target  from  the  pop-up  menu  and  press  the 
“Remove”  button.  Do  this  with  sierra  1  A.  You  will  have  to  select  it  from  the  pop-up 
menu  first. 

=>  When  SierralA  has  been  deleted,  read: 

Also  indicate  own  ship  course  by  selecting  ownship  from  the  menu  and  clicking  to 
represent  the  appropriate  course  vector.  Go  ahead  and  draw  ownship  vector  now. 

=>  When  OS  vector  has  been  drawn,  read 

There  are  instructions  on  the  window  to  remind  you  how  to  do  all  of  this. 

When  you  are  satisfied,  click  the  “Done”  button.  Go  ahead  and  do  this. 

=>  When  GEO  TABLE  window  comes  up,  read 

You  indicate  your  best  guess  for  target  course,  speed,  and  range  estimates  on  this  page. 
You  just  type  values  into  the  appropriate  fields. 

I  want  you  to  provide  information  on  both  of  these  wash-up  displays  for  each  leg  of  the 
scenario,  so  if  there  is  a  leg  you  haven’t  provided  information  on  yet,  press  this  “Next 
Leg...”  button.  If  have  provided  information  on  all  of  the  legs  of  the  scenario,  press  this 
“Done”  button.  This  will  end  the  exercise. 

Wash-up  Free  Practice 

At  this  point,  I  would  like  you  to  practice  using  both  of  these  displays  without  me 
stepping  you  through  it,  so  go  ahead  and  press  the  next  leg  button  and  draw  the  last  leg  of 
the  scenario. 

=>  When  the  GEO  DRAW  window  appears,  read: 

Just  go  ahead  and  place  the  targets  from  the  practice  scenario  on  the  geosit,  making  sure 
to  include  the  course/speed  vector  and  the  range  uncertainty.  Also  plot  the  course  of 
OwnShip.  Try  using  the  “Remove”  button  to  make  sure  you  understand  how  it  works. 
When  you  finish,  press  the  “Done”  button  to  get  to  the  table  screen.  On  the  table  screen, 
enter  numerical  values  to  represent  what  you  drew  on  the  graphical  screen  and  hit  the 
“Done”  button  when  you  are  finished.  If  you  have  any  questions  while  doing  this,  please 
ask. 

=>  Watch  the  subject  interact  with  the  screens  and  answer  any  questions  they  have 

=>  When  they  press  the  “Done”  button  and  get  back  to  the  main  control  screen,  take 
control  of  the  computer  and  press  the  “Stop”,  then  the  “Reset”  button 

=>  You  should  be  at  the  45-minute  point 
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Session  1  data  collection  (30  min.) 

The  subject  is  provided  with  the  same  cover  story  as  the  last  time  (i.e.,  Russia  and 
Ukraine  may  be  out  to  attack  merchant  shipping),  and  also  given  the  specific  status  report 
for  the  first  scenario  -  to  include  OS  course,  speed,  depth,  MERC  bearing.  The  subject 
interacts  with  Ned,  uninterrupted,  until  he  feels  confident  enough  in  the  solution  to  fire, 
or  30  minutes  have  passed.  The  experimenter  plays  a  passive  role,  only  prompting  the 
subject  to  talk  aloud  when  there  is  a  >  10  second  or  so  period  of  silence.  Subjects  will 
complete  both  the  graphical  and  tabular  forms  of  the  hot  wash-up 


=>  Start  the  video  tape  recording 

=>  Read: 

If  you  have  no  further  questions,  we  will  start  the  first  real  scenario. 

=>  Load  the  desired  scenario  by  selecting  it  from  the  pop-up  menu. 

=>  Make  sure  the  “Run  experiment”  check-box  is  still  checked  and  press  the 
“Configure...”  button. 

=>  Enter  the  appropriate  values  in  the  setup  window,  naming  the  data  file  according 
to  the  suggestion  in  the  save  file  dialog.  Make  sure  the  “Collect  Biographical 
Data”  check  box  is  NOT  checked  and  the  “Wash-up”  one  is.  Press  the  “Done” 
button  when  finished. 

=>  Press  the  “Start”  button  to  bring  up  the  problem  description  window 

=>  Read: 


All  scenarios  take  place  with  this  same  basic  problem  description  that  you  saw  in 
practice.  We  will  begin  with  detection  and  run  until  the  point  at  which  you  would  like  to 
initiate  firing  point  procedures.  You  should  indicate  this  by  pressing  the  FPP  button  on 
the  display  navigation  palette.  You  will  then  do  the  hot  wash-up  as  you  did  in  the 
practice  scenario. 

Please  talk  aloud  during  the  entire  problem  -  just  say  out  loud  everything  that  you  sav  to 


yourself  silently. 

You  may  begin  when  ready. 

=>  Watch  passively  as  subjects  complete  scenario.  Interrupt  only  if  they  are  silent 
for  10-15  seconds  or  so,  when  you  do  so,  just  say  “talk  aloud”. 

=>  When  subject  has  completed  the  scenario,  and  gotten  through  the  hot  wash-up, 
take  over  the  computer  and  press  the  “Stop”,  then  “Reset”  button. 

=>  See  if  the  subject  needs  to  take  a  short  break 


= >  You  should  be  at  the  1:15  minute  point. 

Session  2  data  collection  (30  min.) 

Same  as  Session  1  except  that  subjects  work  a  different  scenario,  and  biographical  data  is 
collected 
Read: 

We  will  now  start  the  next  scenario. 
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=>  Load  the  desired  scenario  by  selecting  it  from  the  pop-up  menu. 

=>  Make  sure  the  “Run  experiment”  check-box  is  still  checked  and  press  the 
“Configure...”  button. 

=>  Enter  the  appropriate  values  in  the  setup  window,  naming  the  data  file  according 
to  the  suggestion  in  the  save  file  dialog.  Make  sure  the  “Collect  Biographical 
Data”  AND  “Wash-up”  check  boxes  are  both  checked.  Press  the  “Done”  button 
when  finished. 

=>  Press  the  “Start”  button  to  bring  up  the  problem  description  window 
=>  Read: 

Recall  that  all  scenarios  take  place  under  the  same  basic  conditions.  As  before,  we  will 
begin  with  detection  and  run  until  you  decide  you  are  ready  to  go  to  firing  point 
procedures.  After  the  hot  wash-up,  you  will  also  be  shown  a  screen  containing  some 
biographical  questions. 

Again.  Please  talk  aloud  during  the  entire  problem 
You  may  begin  when  ready. 

=>  Watch  passively  as  subjects  complete  scenario.  Interrupt  only  if  they  are  silent 
for  10-15  seconds  or  so,  when  you  do  so,  just  say  “talk  aloud”. 

=>  When  subject  has  completed  the  scenario,  and  gotten  through  the  hot  wash-up, 
and  biographical  data  screens,  take  over  the  computer  and  press  the  “Stop”, 
then  “Reset”  button. 

=>  See  if  the  subject  needs  to  take  a  short  break 
=>  You  should  be  at  the  1:45  minute  point. 

Experiment  debrief  and  form-filling  (15-25  min.) 

The  goal  here  is  to  examine  recall  of  both  scenarios  and  the  distinctiveness  of  the  recall. 
These  should  be  done  with  paper  and  pencil  and  encourage  the  subjects  to  explain  as  they 
go.  It  might  also  be  interesting  to  collect  data  on  what  the  subjects  made  of  the  simulation 
(reality,  usability,  etc.)  after  they  complete  the  recall  session.  This  can  be  rather  informal. 
Scenario  1  Recall 

=>  Load  the  practice  scenario  by  selecting  it  from  the  pop-up  menu. 

=>  Make  sure  the  “Run  experiment”  check-box  is  still  checked  and  press  the 
“Configure...”  button. 

=>  Enter  the  appropriate  values  in  the  setup  window,  entering  1W  as  the  trial 
number  for  the  first  scenario 

=>  Name  the  data  file  according  to  the  suggestion  in  the  save  file  dialog.  Make  sure 
the  “Collect  Biographical  Data”  check  box  is  not  checked  and  the  “Wash-up” 
one  is.  Press  the  “Done”  button  when  finished. 

=>  Press  the  “Start”  button  to  bring  up  the  problem  description  window 
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=>  Press  the  view  status  report  and  begin  buttons  as  they  come  up,  and  then  press 
the  FPP  button  on  the  control  window  when  it  appears  to  get  to  the  GEO  DRAW 
window 

=>  Read: 

This  is  the  last  portion  of  the  experiment.  I  would  like  you  to  use  both  the  graphical  and 
tabular  wash-up  screens  to  draw  the  events  of  both  scenarios  as  you  did  before.  Begin 
with  the  first  scenario,  drawing  the  targets  on  the  graphical  window  and  then  entering 
numerical  values  in  the  tabular  window.  Be  sure  to  do  this  for  all  targets  and  ownship 
and  for  each  leg.  You  may  begin  when  ready.  When  you  are  finished,  hit  the  Done 
button  on  the  tabular  window. 

=>  When  subject  has  finished  take  over  the  computer  and  press  “Stop”,  then  the 
“Reset”  button. 

Scenario  2  Recall 

=>  Load  the  practice  scenario  by  selecting  it  from  the  pop-up  menu. 

=>  Make  sure  the  “Run  experiment”  check-box  is  still  checked  and  press  the 
“Configure...”  button. 

=>  Enter  the  appropriate  values  in  the  setup  window,  entering  2W  as  the  trial 
number  for  the  first  scenario 

=>  Name  the  data  file  according  to  the  suggestion  in  the  save  file  dialog.  Make  sure 
the  “Collect  Biographical  Data”  check  box  is  NOT  checked  and  the  “Wash-up” 
one  is.  Press  the  “Done”  button  when  finished. 

=>  Press  the  “Start”  button  to  bring  up  the  problem  description  window 

=>  Press  the  view  status  report  and  begin  buttons  as  they  come  up,  and  then  press 
the  FPP  button  on  the  control  window  when  it  appears  to  get  to  the  GEO  DRAW 
window 

=>  Read: 

I  would  now  like  you  to  recount  the  events  in  the  second  scenario.  Be  sure  to  do  this  for 
all  targets  and  ownship  and  for  each  leg.  You  may  begin  when  ready.  When  you  are 
finished,  hit  the  Done  button  on  the  tabular  window. 

=>  When  subject  has  finished  take  over  the  computer  and  press  “Stop”,  then  the 
“Reset”  button. 

Final  Questions 

=>  Transfer  all  files  to  zip  disk  and  label  appropriately 
=>  Hand  the  subject  the  additional  questionnaire. 

=>  Stop  Video  Camera  and  Audio  Recorder. 

Please  take  a  minute  to  fill  out  the  additional  questionnaire. 
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INFORMED  CONSENT 


I  have  been  informed  about  the  purposes  and  procedures  of  this  experiment 
according  to  enclosure  (1)  and  am  willing  to  participate  under  the  conditions 
stated.  I  understand  that: 

(a)  I  may  withdraw  at  any  time  with  no  penalty, 

(b)  All  data  will  be  confidential,  and 

(c)  All  data  will  be  retained  by  the  Principle  Investigator  named  in 
enclosure  (2). 


Print  Name: 
Signature:_ 
Date: _ 
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GENERAL  INFORMATION 


This  is  an  unclassified  system  being  used  for  basic  research.  This 
exercise  will  be  audio/video  recorded.  Please  keep  all  remarks 
unclassified. 

Today  we  will  ask  you  to  participate  in  two  scenarios  using  a  simplified  simulation. 
First  we  will  provide  a  high  level  overview  of  what  you  will  be  doing  and  why  we  are 
asking  you  to  do  it.  When  we  finish  we  will  tell  you  more  specifically  what  we  are 
studying  and  how  your  participation  contributes  to  the  goals  of  the  study. 

Purpose 

The  purpose  of  this  experiment  is  to  gather  data  on  how  command  might  use 
information  that  is  typically  available  in  the  Combat  Control  System.  As  you  know,  the 
new  SSN  that  is  being  planned  will  have  a  dedicated  command  workstation  (CWS)  to 
support  the  Approach  Officer.  As  this  is  a  new  console,  there  is  little  known  about  how 
such  a  workstation  will  be  used.  This  experiment  is  one  in  a  series  of  studies  to  predict 
the  answer  to  that  question.  One  thing  we  do  know,  gleaned  from  the  experience  of  many 
submarine  officers,  is  that  the  CWS  will  require  different  displays  than  those  currently 
available  on  operator  consoles.  Your  participation  will  help  determine  the  content  of 
those  displays. 

Overview  of  the  study 

Your  role  will  be  that  of  the  AO.  Each  scenario  will  begin  with  a  status  report  on  the 
current  situation.  Before  the  first  scenario  begins,  you  will  be  given  practice  with  the 
experimental  procedures  and  with  the  simulation. 

Data  collection  in  this  experiment  depends  on  making  your  information  needs  and 
usage  visible.  A  large  part  of  our  task  is  to  understand  what  information  you  need  and 
when  you  need  it.  Hence,  we  request  that  you  think  aloud,  saying  whatever  comes  to 
mind,  even  if  it  might  be  irrelevant.  You  will  be  given  an  opportunity  to  practice  doing 
this  shortly.  Videotapes  will  be  made  of  every  scenario  and  much  computerized  data  is 
being  collected  as  well.  In  accordance  with  requirements  of  NUWC's  Committee  for  the 
Protection  of  Human  Subjects,  strict  procedures  have  been  established  to  preserve  the 
confidentiality  of  participants.  Outside  of  the  experimenters,  no  one  can  gain  access  to 
information  that  would  reveal  your  identity.  If  you  have  any  concerns  that  we  should 
address,  please  ask  them  now. 
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STUDY  INFORMATION 


Title  of  the  Study: 

Project  NEMO 

Principal  Investigator  (Name,  Code,  Phone): 

Susan  S.  Kirschenbaum,  Ph.D.  Code  2211,  841-4381,  x23835. 

Sponsors: 

Dr.  S.  Dickinson,  NUWC IR  Program  Manager,  and  Dr.  S.  Chipman,  Code  342, 
ONR. 

The  purpose  of  this  work  is  to  study  and  model  the  cognitive  activity  and 
behavior  required  for  situation  assessment.  The  nominal  subject  will  be  the 
submarine  Approach  Officer  (AO).  Situation  assessment  is  a  complex  cognitive 
skill  involving  components  (such  as  information  search  and  assessment,  memory, 
problem  solving,  and  decision  making)  that  have  been  studied  in  isolation,  but 
never  in  the  context  of  event-driven  situation  assessment.  We  expect  our  pursuit  of 
situation  assessment  to  result  in  advances  in  both  theory  and  practice. 

This  experiment  will  take  place  in  the  NUWC  Division  Newport  laboratory 
facility,  the  ARCH  Laboratory  at  George  Mason  University,  or  at  the  participant’s 
own  facility.  It  will  consist  of  subjects  interacting  with  staged  scenarios  using  a 
simplified  simulation  created  for  this  experiment  and  hosted  on  a  Macintosh. 
Subjects  will  be  doing  a  task  that  is  similar  to  their  normal  job  including  assessing 
data  and  giving  orders  for  the  control  of  the  simulated  submarine  in  a  specified 
mission.  They  will  each  have  two  scenario  segments.  The  total  experimental  time 
will  be  approximately  two  hours.  All  sessions  will  be  video  taped 

This  experiment  is  expected  to  require  10  to  15  subjects  in  each  of  three 
groups,  novices  (prior  to  first  deployment),  intermediates  (after  two  to  four  years  at 
sea),  and  experienced  (Executive  Officer  or  more  experience).  Subjects  will  be  run 
individually,  at  the  convenience  of  the  subjects.  No  deception  will  be  employed. 
The  standard  informed  consent  form  will  be  used  with  a  description  of  the 
experiment.  Data  recording  will  include  computer  interactions,  videotapes  of  the 
screen,  audio  recordings  of  think-aloud  protocols,  and  written  questionnaires.  All 
records  will  be  retained  by  the  P.I.  No  data  will  be  identified  by  subject.  Data  will 
be  analyzed  by  statistical  and  process-tracing  methods,  including  cognitive 
modeling  and  other  appropriate  methods.  Subjects  will  be  able  to  request  copies  of 
resulting  publications  and  will  be  given  time  at  the  end  of  sessions  to  ask 
questions. 

The  P.I.  of  this  project  is  an  experimental  psychologist  and  agrees  to  conduct 
the  experiment  under  conditions  recommended  by  the  American  Psychological 
Association  and  approved  by  this  committee. 
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APPENDIX  C:  EXPERIMENT  KEY 
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Table  C4.2.  Timeline  for  60  Minute  Scenarios  (course  and  speed  remain  the  same  as  initial  values  above  unless  specified). 


Figure  C4.1.  Scenario  Cl  Geometry. 


zl  becomes  High  Dby 
on  right,  drawing  ft 
Course,  speed,  By,  Range 
OS  000,  9 

Merch334, 12,  113,15000 
Sub  184,10,043,  7000 


Figure  C4.2.  Scenario  Zl  Geometry. 


Figure  C4.3  Scenario  C2  Geometry. 


New  22,  High  Dby 
on  left,  drawing  left 
Course,  speed,  By,  range 
OS  225,  9 
Merch  160,  12,  160,15000 
Sub  350,  10, 150,  7000 


Figure  C4.4  Scenario  Z2  Geometry. 
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Trial  Order 

Subject 

Experience  level 

Trial  1 

Trial  2 

SOI 

Novice 

z2 

c2 

S02 

Novice 

z2 

cl 

S03 

Novice 

c2 

z2 

S04 

Novice 

zl 

cl 

S05 

Novice 

cl 

zl 

S06 

Novice 

c2 

zl 

S07 

*r  Novice 

zl 

c2 

S08 

Novice 

cl 

z2 

S09 

Intermediate 

z2 

c2 

S10 

Intermediate 

z2 

cl 

Sll 

Intermediate 

c2 

z2 

S12 

Intermediate 

zl 

cl 

S13 

Intermediate 

cl 

zl 

S14 

Intermediate 

c2 

zl 

S15 

Intermediate 

zl 

WEM 

S16 

Intermediate 

cl 

z2 

S17 

Expert 

z2 

c2 

S18 

Expert 

z2 

cl 

S19 

Expert 

c2 

z2 

S20 

Expert 

zl 

cl 

S21 

Expert 

cl 

zl 

S22 

Expert 

c2 

zl 

S23 

Expert 

zl 

c2 

S24 

Expert 

cl 

z2 

Table  C4.3  Latin  Square  Design. 


APPENDIX  D:  ADDITIONAL  QUESTIONABLE 

ADDITIONAL  QUESTIONNAIRE 

DATE: _ 

INTERVIEWER: _ 

SUBJECT: _ 

1 .  Could  you  tell  me  what  you  thought  of  the  situation? 

2.  Was  the  situation  “realistic”  enough  for  you?  Any  comments? 

1  Very  Unrealistic 

2  Unrealistic 

3  Borderline 

4  Realistic 

5  Very  Realistic 

3.  How  difficult  were  the  scenarios?  (Please  provide  a  score  for  each  run)  Any 
comments? 

1  Very  Difficult 

2  Difficult 

3  Borderline 

4  Easy 

5  Very  Easy 

4.  Please  rate  the  effectiveness  of  each  display. 

1  Very  Ineffective 

2  Ineffective 

3  Borderline 

4  Effective 

5  Very  Effective 

LOS _ OSC _ SVP _ 

TMA _ TAPBB _ TAPNB _ SAPBB _ 

TF _ TB _ GEOSIT _ 


5.  Please  rank  the  usefulness  of  the  displays.  One  for  the  most  useful,  10  for  the  least. 
No  ties  allowed. 

LOS _ OSC _ SVP _ 

IMA _ TAPBB _ TAPNB _ SAPBB _ 

TF _ TB _ GEOSIT _ 
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6.  The  ultimate  goal  of  the  project  is  to  provide  for  the  design  of  the  CWS  of  the  NSSN. 
If  you  were  in  charge,  what  kind  of  design  would  you  make?  Include  any  and  all 
displays/information/graphical  interfaces  that  you  desire. 

(Use  Blank  sheets  provided) 

7.  Any  more  comments? 
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APPENDIX  E:  ASSUMED  TORPEDO  CHARACTERISTICS 


Name: 

Manufacturer: 

Diameter: 

Length: 

Weight 

Range: 

Speed: 

Operating  Depth: 
Acoustic  acquire  range: 

(Friedman,  1983) 


MK-48  Heavyweight  Torpedo 

Gould  Electronics 

21  inch 

230  inch 

3480  pounds 

35000  yards 

55  knots 

2500  feet 

2000  yards 
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APPENDIX  F:  VB  MARCO  CODE 


Option  Base  1 

Sub  doall  ( ) 

Call  breakoutdata 
Call  displayanalysis 
End  Sub 

Sub  breakoutdata ( ) 

Dim  newdata,  newtruth,  datcount,  truthcount 
datcount  =  0 
truthcount  =  0 

Set  newdata  =  ActiveWorkbook. Worksheets  .Add 
Set  newtruth  *  ActiveWorkbook. Worksheets .Add 
newdata. Name  =  "Data" 
newtruth. Name  =  "Truth" 

For  Each  rw  In  Sheets (3) . Rows 

If  rw. Cells (1)  =  ""  And  rw.Row  >  1  Then 
Exit  For 
Else 

If  rw. Cells (1)  =  ("TRU")  Then 

truthcount  =  truthcount  +  1 
newtruth. Rows (truthcount) .Value  =  rw. Value 
Elself  rw.Row  >  6  Then 

datcount  =  datcount  +  1 

newdata. Rows (datcount) .Value  =  rw. Value 

End  If 
End  If 
Next  rw 

Call  cleant ruth (newtruth) 

Call  cleandata (newdata) 

End  Sub 

Sub  cleantruth (truthsheet) 

For  N  =  1  To  5 

truthsheet. Columns (1) .Delete 
Next  N 

truthsheet . Columns (2) .Delete 

truthsheet .Columns (1) .Number Format  =  "h:mm:ss" 

End  Sub 

Sub  cleandata (datasheet) 

For  N  =  8  To  20 

datasheet . Columns (8) .Delete 
Next  N 

With  datasheet 

. Columns (6) .NumberFormat  =  "h:mm:ss" 

. Rows ( 1 ) . Font . Bold  =  True 
.  Columns  ( "A:  BW" )  . AutoFit 

.Range ("A: BW")  .Sort  key 1 :  =  . Columns ( "E" ) ,  header :=xl Yes 
End  With 

datasheet . Activate 
ActiveWindow. SplitRow  =  1 
ActiveWindow. FreezePanes  -  True 
End  Sub 
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*  Master  function  to  run  the  analysis  by  display 
Sub  displayanalysis ( ) 

Call  displayaccess summary 
Call  displaysequence 
Call  addfreqchart 
Call  addtotchart 
Call  addavgchart 
Call  repositioncharts 
End  Sub 

Sub  displayaccesssummary { ) 

Dim  countsheet  As  Variant 

Dim  discount,  rowcount,  oldtime,  newtime  As  Integer 

Dim  fpctstr,  tpctstr  As  String  'sumstr,  avgstr,  stdevstr. 

Dim  labarray,  disarray,  timearray ( )  As  Double 

disarray  =  Array ( "GEOS IT",  "LOS",  "OSC",  "SONAR-BB- SPHERE", 

"SONAR-BB-TOWED",  "SONAR-NB-TOWED",  "SVP",  "TBR",  "TFP",  "TMA" )  . 

labarray  =  Array ( "Display",  "Frequency",  "%  Frequency",  "Tot  Time", 
"%Time",  "Avg  Time",  "Std  Dev") 
rowcount  =  1 

Set  countsheet  =  ActiveWorkbook. Worksheets  .Add 
Call  prepcountsheet ( countsheet,  labarray) 

For  Each  d  In  disarray 
discount  =  0 
oldtime  =  0 
ReDim  timearray (1) 

For  Each  rw  In  Worksheets ( "Data") . Rows 
If  rw. Cells (1)  =  ""  Then 

rowcount  =  rowcount  +  1 

'sumstr  =  "=sum(R"  &  rowcount  &  "C8"  &  " :R"  &  rowcount  & 

"C"  &  UBound (timearray)  +  6  &  ")" 

'avgstr  =  "=average  (R"  &  rowcount  &  "C8"  &  "  :R"  &  rowcount 
&  "C"  &  UBound  (timearray)  +  6  &  ")" 

'stdevstr  =  "=stdev(R"  &  rowcount  &  "C8"  &  ":R"  &  rowcount 
&  "C"  &  UBound (timearray)  +  6  &  ")" 
fpctstr  =  "=(R"  &  rowcount  &  "C2"  &  "/"  &  "R12C2) " 
tpctstr  =  "=(R"  &  rowcount  &  "C4"  &  "/"  &  "R12C4)" 

With  countsheet 

.Range ( .Cells (rowcount,  8),  . Cells (rowcount,  discount  + 

7)  )  . FormulaArray  =  timearray. Cells  (rowcount,  1)  .Value 
=  d. Cells (rowcount,  2) .Value  =  discount 
. Cells (rowcount,  3) .FormulaRICl  =  fpctstr 
'  .Cells  (rowcount,  4)  . FormulaRICl  =  sumstr 

. Cells (rowcount,  5) . FormulaRICl  =  tpctstr 
'  . Cells (rowcount,  6) . FormulaRICl  =  avgstr 

'  .Cells (rowcount,  7) . FormulaRICl  =  stdevstr 

.Cells (rowcount,  4) .Value  =  mysum( timearray) 

. Cells (rowcount,  6) .Value  =  myaverage (timearray ) 

.Cells (rowcount,  7)  .Value  =  mystdev{ timearray) 

End  With 
Exit  For 

Elself  rw.  Cells (2)  =  "DISPLAY- CHANGE"  And  rw. Cells (3)  =  d  Then 
discount  =  discount  +  1 
newtime  =  rw. Cells (5) .Value 
ReDim  Preserve  timearray  (UBound  (timearray)  +  1) 
timearray  (discount)  =  newtime  -  oldtime 
oldtime  =  newtime 
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Elself  rw. Cells (2)  =  " DISPLAY- CHANGE"  Then 
oldtime  =  rw. Cells (5) .Value 
End  If 
Next  rw 
Next  d 
End  Sub 

Sub  prepcountsheet (sheet,  labelarray) 

With  sheet 

.Name  =  "Displays” 

. Range ( "A1 : Gl") . FormulaArray  =  labelarray 
.Range ("A1:G1") .Font. Bold  =  True 
. Cells (12,  1) .Value  =  "Totals" 

.Cells(12,  1} .Font. Bold  =  True 
. Cells ( 12 ,  2)  .FormulaRICl  =  "=sum(R2C2 : R11C2 ) " 

. Cells (12,  4) .FormulaRICl  =  ”=sum(R2C4 : R11C4 ) " 

. Range ("Al : Gil”) .Borders. LineStyle  =  xlContinuous 
End  With 
End  Sub 

Function  mysum(anarray) 

For  Each  el  In  anarray 
my  sum  =  my  sum  +  el 
Next  el 
End  Function 

Function  myaverage (anarray) 

If  UBound( anarray)  >  1  Then 
myaverage  =  mysum( anarray)  /  (UBound (anarray)  -  1) 

Else 

myaverage  =  0 
End  If 

End  Function 

Function  mystdev( anarray) 

Dim  sm,  sqrd,  N  As  Integer 
N  =  UBound (anarray)  -  1 
For  Each  el  In  anarray 
sm  =  sm  +  el 
sqrd  =  sqrd  +  el  A  2 
Next  el 
If  N  >  1  Then 

mystdev  =  Sqr (( (N  *  sqrd)  -  sm  A  2)  /  (N  *  (N  -  1)  )  ) 

Else:  mystdev  =  0 
End  If 

End  Function 

Sub  displaysequence ( ) 

Dim  labarray,  disarray,  rowcount,  oldtime,  newtime 
disarray  =  Array ( "GEOS IT",  "LOS",  "OSC",  " SONAR- BB- S PHERE " , 
"SONAR-BB-TOWED",  "SONAR-NB-TOWED",  "SVP",  "TBR" ,  "TFP",  "TMA" ) 
labarray  =  Array ( "Time",  "Display",  "Duration") 
rowcount  =  16 
For  Each  d  In  disarray 
oldtime  =  0 

For  Each  rw  In  Worksheets ( "Data" ). Rows 
If  rw. Cells (1)  =  ""  Then 
Exit  For 

Elself  rw. Cells (2)  =  " DISPLAY- CHANGE"  And  rw. Cells (3)  = 


d  Then 
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newtime  =  rw. Cells  (5)  .Value 
rowcount  =  rowcount  +  1 
With  Worksheets ("Displays” ) 

. Cells (rowcount,  1) .Value  =  newtime 
. Cells (rowcount,  2) .Value  =  d 
.Cells (rowcount,  3) .Value  =  newtime  -  oldtime 
End  With 

oldtime  =  newtime 

Elself  rw. Cells (2)  =  "DISPLAY- CHANGE"  Then 
oldtime  =  rw. Cells (5) .Value 
End  If 
Next  rw 
Next  d 

With  Worksheets ( "Displays") 

. Range ( "A15 : C15 " ) . FormulaArray  =  labarray 

.Range ("A15: CIS") . Font. Bold  =  True 

.Range ( "A16 : C1000" ) .Sort  keyl :=. Columns ("A") 

.Columns ("A: BW") .AutoFit 
End  With 
End  Sub 

Sub  addf reqchart ( ) 

Charts .Add 

ActiveChart . ChartType  =  xlColumnClustered 
ActiveChart . SetSourceData 

Source :=Sheets ("Displays") . Range ( "A1 : All, Cl : Cll" ) ,  _ 

PlotBy : =xlColumns 

ActiveChart . Location  Where : =xlLocationAsOb j ect,  Name : ="Displays " 
With  ActiveChart 

.  HasTitle  =  True 

. ChartTitle . Characters . Text  =  "Display  Frequency" 

.Axes (xlCategory,  xlPrimary) .HasTitle  =  False 
.Axes (xl Value,  xlPrimary) .HasTitle  =  True 

.Axes (xlValue,  xlPrimary)  .AxisTitle . Characters . Text  =  "% 
Frequency" .HasLegend  =  False 
End  With 
End  Sub 

Sub  addtotchart { ) 

Charts .Add 

ActiveChart. ChartType  =  xlColumnClustered 
ActiveChart . SetSourceData 

Source:=Sheets ("Displays") .Range ("A1:A11,E1:E11") ,  _ 

PlotBy :=xlColumns 

ActiveChart . Location  Where : =xlLocationAsOb j  ect.  Name : ="Displays " 
With  ActiveChart 

.HasTitle  =  True 

.ChartTitle. Characters. Text  =  "Display  Time" 

.Axes (xlCategory,  xlPrimary) .HasTitle  =  False 
.Axes (xl Value,  xlPrimary) .HasTitle  =  True 

.Axes  (xl Value,  xlPrimary)  .AxisTitle .  Characters  .Text  =  "%  Time" 
.HasLegend  =  False 
End  With 
End  Sub 
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Sub  addavgchart ( ) 

Charts .Add 

ActiveChart.ChartType  =  xlColumnClustered 
ActiveChart . SetSourceData 

Source :~Sheets ("Displays") . Range ( "A1 : All , FI : Fll" ) ,  _ 
PlotBy:=xlColumns 

ActiveChart. Location  Where : =xlLocationAsOb ject,  Name :="Di splays" 
With  ActiveChart 

.HasTitle  =  True 

. ChartTitle. Characters. Text  =  "Average  Display  Time" 

.Axes (xlCategory,  xlPrimary) .HasTitle  =  False 
.Axes (xlValue,  xlPrimary) .HasTitle  =  True 

.Axes  (xlValue,  xlPrimary) .AxisTitle . Characters . Text  =  "Time 
(secs ) " . HasLegend  =  False 
End  With 
End  Sub 

Sub  repositioncharts ( ) 

ActiveSheet. Shapes ("Chart  3") . IncrementLeft  496# 

ActiveSheet. Shapes ("Chart  3") . IncrementTop  -8# 

ActiveSheet. Shapes ("Chart  2") .IncrementLeft  81# 

ActiveSheet . Shapes ("Chart  2") .IncrementTop  387# 

ActiveSheet. Shapes ("Chart  1") . IncrementLeft  69# 

ActiveSheet . Shapes ( "Chart  1" ). IncrementTop  -8# 

End  Sub 
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APPENDIX  G:  OPTIMIZATION  FORMULATION  AND  CODE 

A.  NPS  FORMULATION 

For  each  trial  i,  GAMS  is  used  to  determine  the  closest  point  of  intersection  of  the 
submarine  and  torpedo  trajectories,  if  the  intersection  occurs  before  TMAX.  Otherwise, 
it  determines  the  range  at  TMAX. 

Problem:  For  the  CPA  parameters  (Time,  Range)  for  the  weapon  and  the  target. 

INDEX:  i  TRIALi=l..I 

Data:  Xsoi  initial  submarine  x  position  for  trial  i 

Yso;  initial  submarine  y  position  for  trial  i 

Xto  initial  torpedo  x  position  for  trial  i 
Yto  initial  torpedo  y  position  for  trial  i 
Vsxi  submarine  velocity  in  the  x  direction  for  trial  i 
Vsyi  submarine  velocity  in  the  y  direction  for  trial  i 
VtXi  torpedo  velocity  in  the  x  direction  for  trial  i 
Vtyi  torpedo  velocity  in  the  y  direction  for  trial  i 
Tmax  maximum  run  time  of  torpedo  in  seconds 

Variables:  Rhi  CPA  range  for  trial  i 

Ti  CPA  time  for  trial  i 

Formulation  for  trial  i: 

min  Rhi 

s.t  Rhi=sqrt(((Xso+Vsx*Ti)-(Xto+Vtx*Ti))A2+ 

((Yso+Vsy*  Tj  )-(Yto+Vty*  Ti  ))A2) 

0<=Ti<+Tmax 

Output:  Rhi 

Ti 

B.  GAMS  CODE 

*  DAVID  SOLDOW 

*  CPA  OPTIMIZATION  ROUTINE 

$Title:  CPA  optimization  routine 

SETS 

I  /1*65/ 

PARAMETERS 

Xso(I)  /I  2881.801, (EDITED  FOR  SPACE)  65  2116.524/ 

Yso (I)  /I  -4971.37, (EDITED  FOR  SPACE)  65  437.9259/ 

Vsx(I)  /I  -5.16831, (EDITED  FOR  SPACE)  65  -1.31678/ 

Vsy(I)  /I  -1.88111, (EDITED  FOR  SPACE)  65  -3.61782/ 

Vtx(I)  /I  11.33185, (EDITED  FOR  SPACE)  65  29.35145/ 

Vty(I)  /I  -28.0473, (EDITED  FOR  SPACE)  65  7.318137/; 

SCALAR  TMAX  /1145.5/; 

SCALAR  Xto  /0/; 
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SCALAR  Xto  /0/; 

SCALAR  Yto  /0/; 

Scalars 

Xsoi,  Vsxi,  Vtxi,  Ysoi,  Vsyi,  Vtyi  ; 

VARIABLES 

Z  CPA  RANGE  FOR  TRIAL  I 

T  CPA  TIME  FOR  TRIAL  I  ; 

T . LO=0 ; 

T.UP=TMAX; 

EQUATIONS 

CPA  OBJECTIVE  VALUE  FOR  TRIAL  i; 

CPA  . .  Z=e=SQRT (sqr ( (XsoI+VsxI*T) - (Xto+VtxI*T) ) 
+sqr ( (YsoI+VsyI*T) - (Yto+VtyI*T) ) ) ; 
MODEL  SHOTCPA  /ALL/; 

FILE  REPORT/ SHOTCPA. RPT/ ; 

PUT  REPORT;  REPORT. ND=5; 

PUT  /'TRIAL  RH (I)  T(I)'; 

PUT  /’ - '; 

LOOP (I, 

Xsoi=xso (i) ; 

Vsxi=vsx (i) ; 

Vtxi=Vtx (i) ; 

Ysoi=Yso  (i) ; 

Vsyi=vsy(i) ; 

Vtyi=vty (i) ; 

SOLVE  SHOTCPA  USING  NLP  MINIMIZING  Z; 

PUT  I.tl; 

PUT  Z . 1 ; 

PUT  T.l; 

)  ; 

PUTCLOSE  REPORT; 


C.  OUTPUT 

TRIAL  RH(I)  T (I ) 


1  214.395  186 

2  214.5673  124 

(EDITED  FOR  SPACE) 

65  298.5145  66 
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